Sometime ago, I finished reading the Advanced React book by Nadia Makarevich. The book is quite popular as an advanced React resource and might be on your wishlist as well.

The book is packed with advanced knowledge and practical tips, so if you haven’t heard about it yet, I suggest adding it to your readlist.

Who is this book for?

The book is for anyone seeking to elevate their knowledge and skills in React beyond introductory concepts.

However, before picking it up, you need a working understanding of most React concepts, such as reconciliation, React Hooks, component lifecycle methods, and Rendering, among others.

As the book aims to convey advanced knowledge, having the basics covered is essential.

Having working understanding a pre-requisite to read this book.

What makes this book stand out?

I have read other books on React as well. But most of those for beginners are too niche, with less focus on advanced concepts.

Here, Nadia justifies the book’s title by revealing knowledge one might not have known.

Subscribe now

The topics this book covers.

The book covers most essential React concepts that developers use regularly. Some of the topics are:

1. Re-renders

2. Diffing and Reconciliation

3. Memoization

4. Refs

5. React Context

6. HOCs

7. Closures

8. uselayoutEffect

9. Some tips on data fetching WRT race conditions

and some more.

Let’s review the book.

1. Going in-depth after sharing common myths and concepts

This is the best thing about this book. Nadia first shares common myths and incorrect mental models related to the topic. Then the idea is contradicted and shown not to be always true.

I found this approach better than simply explaining the concepts as they are. This way, one can first go through some misconceptions, then see why those are wrong mental models and what the correct approach is.

Some of the myths explained in the books concern memoisation, state management, and rendering, among others.

2. Practical tips with code snippet

For each topic and solution, the author shares relevant code snippets of components which might help improve your understanding.

So it’s not just theory but also code snippets to get your hands dirty.

3. The book packs knowledge in one place

The knowledge this book provides requires a alot of research and experience.

Personally, I had explored tens of articles and blogs to understand different topics, but the book consolidates all of that in one place.

So, it’s a good way to save time and learn what is essential.

4. Language is easier to understand

If you can't explain it simply, you don't understand it well enough. Albert Einstein

Although the book communicates the advanced concepts, the language is quite simple.

It doesn’t use any jargon or hard terms to make an impression.

Instead, all the ideas are discussed in basic English.

5. Some concepts might seem dated

Although the book conveys foundational concepts and aims to reestablish mental models, some ideas, such as render props, are no longer used commonly.

But you can read the chapters of what you are working with.

6. The code snippets are slightly opinionated

As mentioned earlier, the book provides the code snippets for each tip, trick and concept. However, sometimes the author provides their own code snippet and explains how to handle the solution.

But it’s my personal opinion: once you understand the problem & solution, you can try creating your own.

Conclusion

Overall, the book is one of the best reads if you want to upgrade your understanding of React.

So, you are a backend engineer who wants to get into frontend but can’t figure out how?

Continue reading, as this post is for you.

The frontend is not just about creating buttons or layouts using React or another framework; it also involves understanding user experience, building UI, optimising performance, and more.

To simplify, the frontend development is broadly divided into

1. UI/UX developers (building complex UI/UX layouts with animations, etc.)

2. Frontend Developer (mostly around building logic handling)

Based on my experience, backend developers who transition to the frontend tend to be more inclined towards the logical aspects than UI/UX development.

Most of the companies building complex frontends have dedicated UI/UX developers who focus on building complex layouts.

Irrespective of which path you follow, you will have to grasp the same basics.

Software Visualized

How to transition from frontend to backend development: A complete roadmap.

You are a front-end developer who wants to learn and build backend systems and become a full-stack developer…

Read more

a year ago · 12 likes · Ali Raza

Basics of Frontend Development

HTML, CSS, and JavaScript

You can’t escape these three regardless of what you are working on.

1. HTML & CSS

You can take a crash course on these topics and try to understand the basic tags, elements, HTML syntax, meta tags, and other key concepts.

In CSS, it is essential to learn about the Box Model, positioning, layout styling (such as flex and grid), transitions, media queries, and pseudo-elements.

Don’t start with Tailwind CSS directly, but first learn Vanilla CSS.

2. JavaScript

JS is the language of the frontend, and browsers only understand this. Unlike Backend, you can’t skip it.

Although JavaScript is vast and encompasses many concepts, we will limit our discussion to the concepts necessary for frontend development.

1. Data Structure

2. Hoisting

3. Closures

4. DOM Manipulation

5. Arrow & normal functions

6. Event Loop

7. Scope and this keyword.

8. Web APIs

9. Promises, async, and await

10. Prototype Inheritance

11. Debugger

12. Console

13. Security: XSS, inline scripts, CSRF, input validation and sanitisation, Content Security Policy, authentication and access controls

Once you understand the theory and syntax, it’s time to build some projects.

1. Chat Bot App

2. Quiz App

What to learn more?

1. Optimization

2. Browsers

3. Accessibility

Subscribe now

Frameworks & Libraries

React and Next.js are widely used frontend homework’s and I will recommend start by learning these important concepts.

React

• Philosophy of React and reusable components

  • Imperative vs Declarative approach

  • Reusable components

  • Representational components

• Client-side vs server-side rendering

• Component lifecycle methods

• Hydration

• Hooks and custom hooks (useState, useEffect

• State Management (useState, Zustand)

• React Query, React Router and other libraries.

• Devtools such as Vite, Webpack, React DevTools, etc.

Next.js

• Server-side rendering

• Static-site generation

• Incremental Static Regeneration

• App & folder structure, especially if building full-stack apps

• Next.js routing

• Data fetching methods

• Optimisation, such as images.

• server and client components and server actions

• Authentication with NextAuth

Building Projects

It’s recommended that you build both full-stack and client-side apps on Next.js and React, respectively. It will help you master concepts in a better way.

• Your favourite SAAS application with CRUDs

• Jira clone

• Facebook app components

• Substack webapp clone with CRUDs

Practice & practice. Once you understand these concepts, you can build different projects to practice and learn more.

As you are mainly a Backend developer but want to learn frontend development, these topics are enough for you to learn building decent frontend apps.

Mobile and Desktop Apps

If you master this path, it will help you learn to build hybrid mobile and desktop apps as well.

1. React Native - mobile apps (not my recommended framework)

2. Electron - Desktop apps

So, you will be able to utilise your learnings in a different domain and stack and build more solutions.

Focusing on UI/UX

If you are fascinated by beautiful websites with cool animations and eye-catching designs, you will focus more on a different set of technologies.

UI/UX development is front of frontend development.

Even for UI/UX, the HTML, CSS and JS requirements are the same, but they focus more on implementing UI/UX instead of handling logic, API calls, etc.

So, they specialised in a different skillset.

Skills

1. Building pixel-perfect designs

2. Responsive layouts

3. Accessibility

4. Design tools such as Figma, Adobe XD.

5. Animation libraries such as GSAP, three.js, anime.js, etc.

6. Mastering HTML/CSS

Now, these are additional skills one needs to learn. Even if you are not so well-off in JavaScript and logic building, the above-mentioned skills are more important to build amazing UI/UX.

Projects

There is a good portfolio published on GSAP and other animation libraries to take inspiration from and build. You can go through that to build a project or learn some new styles and designs.

You can go through these links.

1. GSAP Showcase

2. Threejs Examples

Summary

I really appreciate your willingness to get into customer experience and understand how decisions on the backend actually show to the end-users.

I recommend that you follow the frontend development path instead of UI/UX to learn more and scale your skillset.

Good luck!

Subscribe now

If you browse Netflix, you will notice that it provides a seamless and faster experience.

The homepage loads swiftly; movie suggestions appear almost instantly. Clicking on a title brings up the details without delay, and playback begins in a second.

All these things work because the Netflix team carefully planned the frontend system, API schema, image compression and formats, and structure.

We will discuss these practices and system design, taking notes on how to implement them in other applications.

API

GraphQL

Netflix uses GraphQL APIs to fetch data from servers. The structure for each API is well defined.

There are APIs such as

• Movie listing API

• Movie details API

• Token APIs

The APIs’ latency is within the milliseconds range and likely loads from the cache.

It demonstrates the scale of system design, the load those servers have, and how they deliver quickly.

Netflix Frontend System Design: Strategies and learnings

This section will examine how Netflix built its frontend system to provide an excellent experience.

Pre-fetching

In pre-fetching data is downloaded/fetched before the user performs an action or reaches that portion.

The Netflix frontend mostly prefetches data. It doesn't always wait for user actions to fetch the required data. Instead, it anticipates and performs in the background.

Subscribe now

Initial Page Load

When the user reaches the profiles section, the website fetches data for the initial section of the homepage. The data usually includes

1. Thumbnail URLs

2. Title and relevant info

3. categories, etc.

So, when you visit your profile section, the Netflix website has required data available to show to the user when they visit the next homepage of their profile.

Hovering over a title

As mentioned earlier, Netflix prefetches some data required for the initial view but loads additional data per the user's action.

Now, when you hover over an image, it will

1. Download a quick view video of the title

2. Download extra information for the hover box and details modal

3. Fetch the suggested movies’ IDs

The first two actions provide data for the hovered view, and if the user clicks on the modal, it will show the details for that movie/series.

However, it only fetched the movie IDs of the suggested one, and fetched relevant info only when the user opened the details of the movie.

This is a better choice because the user opening the details modal is optional, and the movie detail API's bandwidth and latency are reduced.

Once the details model is opened, as shown in the image above, it will load the movie trailer or image and related info, and download data for suggested titles as per the movie IDs fetched in the previous steps.

Learning

1. Design UI/UX and APIs to show minimum details first if there are too many items on the listing

2. If the user hovers over or loads an item, load the additional data.

3. For further peripheral items, such as suggested listings, fetch data on demand only.

4. The API's latency should be double-digit milliseconds at max to load and show content quickly.

   Lazy loading

The Netflix app doesn't load all sections at once.

Instead, when a user scrolls down, different titles against each category are shown as per the categories set by the system based on the user's history or preference.

As expected, the data for the next categories is fetched as the user scrolls down, and roughly the next 5-6 category sections are already preloaded with that data.

Here, Netflix provides a seamless UX and demonstrates effective API and data management.

Learning

1. Loading all data simultaneously might slow the rendering and impact the experience.

2. So, always fetch and show relevant data only on the frontend and load more as the user scrolls.

3. The APIs should be designed to return data in chunks faster.

Images

The images are heavily compressed as per the place.

For thumbnails, the average size is 20-30 KB.

The images shown during the sneak peek are almost 0 bits due to the blob structure being used.

The reasons are quite logical.

The thumbnail size is small, around 300 by 160, and mostly the title and a picture of the user's preference are shown.

Here, showing high-quality images comprising hundreds of KBs doesn't make sense.

It will slow down the loading because hundreds of titles are shown simultaneously.

Learning

1. If you have an e-commerce store or some app where images are shown,

2. Compress and create images as per different view sizes

3. Always load minimal-size photos for the first view, such as on product listing pages.

4. If the user hovers and the size of the viewport changes, only then will a higher quality image be loaded.

5. By doing this, you will be

   1. loading the website faster

   2. offering a better experience

   3. Saving the customer's internet data

Conclusion

I hope you have learnt how to build performant frontend systems at scale to serve millions of users and maintain consistency.

Subscribe now

I am writing a step-by-step guide with topics at each level and projects you should build instead of bombarding you with backend terms and topics.

By following this guide, you will know the important topics at each stage, such as beginner and medium level, and what projects to build to implement theoretical concepts.

At my job, I have trained tens of interns and transitioned front-end developers into back-end engineers.

So, how do we do it?

1. I will first enlist the topics you need to learn and the subtopics. You can use YT boot camps, courses, and books of your own choice, but I have shared my suggestions at the end.

2. A project with implementation will be mentioned at the beginner and intermediate levels so that you know what to build in the beginning.

3. New and slightly advanced concepts will be introduced gradually so that you can learn those and implement them in existing projects.

Roadmap

You first need to learn some basic concepts related to backend development. These concepts will remain the same irrespective of the programming language or framework.

General Backend Concepts

1. HTTP/HTTPS

2. SSL/TLS Encryption

3. TCP vs UDP

4. How does the server work?

Overview of APIs

1. What are RESTful APIs?

2. CORS

3. HTTP methods.

4. Status codes

5. Idempotency

6. OpenAPI Specification

7. API payload validation

8. API Versioning

Security

1. Concept of Authentication and Authorization

2. OAuth 2.0: overview, what it is, and how it works?

3. JWT: What it is and how it works?

4. Understanding of Session Cookies

5. HTTP-only cookies

6. Encryption

7. Hashing

By now, you have understood some concepts. Now, let’s move to Node.js to get the hands dirty.

Why choose Node.js to write the first backend code?

As you are already familiar with JavaScript, it will be much easier and convenient to implement the concepts you learn across.

If another programming language is chosen, it will take you some time to learn the syntax and concepts. So, we will skip all those steps here.

Node.js essentials concept

1. Basics of Node.js

2. Spinning up a Node.js server

3. Create an API endpoint, send some dummy JSON responses, and call your first API from Postman.

Congratulations. You have achieved something meaningful now.

Let’s now move to databases.

Database

1. Basics of DB, what it is, etc.

2. Pick MongoDB or PostgreSQL and read a little bit about it.

3. Learn concepts of schemas, relationships, unique constraints, indexing, etc.

4. Overview of commands for operations such as insert, update, delete, and get and play with these in the database console.

5. Add the database to your project; add services for CRUD operations.

Create a basic user management service

Now, we will build a minimal user service to implement the authentication and authorization, encryption, and hashing concepts we have learned.

1. Install JWT in the project.

2. Add an API for creating users and hash passwords and save them in the database.

3. On login, return the JWT.

For read, update, and delete endpoints, create a middleware and verify the JWT.

Build a task to-do list project

1. Design a DB model/schema for your project without taking help from the internet for objects, at least.

2. Create CRUD endpoints for to-do list items and add them to the database.

3. Implement middleware for payload validation and authorization.

4. Implement sign-up and sign-in support.

Congratulations! You have built the basic backend of an app that can be integrated with the frontend.

Before moving ahead, build a lot of APIs using the stuff you have learned so far. Some of the projects are:

1. URL shortener

   1. CRUD APIs.

   2. Dashboard for analytics

2. Expense tracker

After building these projects, I hope that you have fully grasped the basic concepts.

Subscribe now

Now, we will move ahead and learn some new concepts on top of existing knowledge.

APIs

1. GraphQL APIs and comparison with RESTful ones.

2. Rate-limiting and throatling-related concepts and implementations

3. Pagination & filtering

4. RBACs: Role-Based Access Controls

Architectural Patterns

1. concurrency, parallelism and Multi-threading

Caching

1. Concepts of caching

2. Strategies such as Cache-Aside, Write-Through, Write-Behind, Read-Through

3. Redis: data structures and their usage

4. Cache eviction strategies

Database

1. ACID properties

2. Transactions

3. Query Optimizations

4. ORMs

Real-time data

1. Server-Sent Events

2. Websockets

3. Polling

4. Webhooks

Monitoring & Logging

1. APM: Overview

2. Monitoring & SLAs: What are 999999s, etc?

3. Explore Sentry and other open-source tools such as Grafana and Prometheus.

Cloud & Networking

1. IP Addressing & Subnetting (IPv4, IPv6)

2. DNS Basics (How domain names resolve)

3. Sockets & Ports (How services communicate)

4. Firewalls & NAT (Basic security and network translation)

5. Network Performance Optimization (Keep-Alive, Pipelining, QUIC)

Deploying the App

1. Overview of CI/CD process

2. Basics of docker

   1. Image creation

   2. Running docker container

3. Deploying on

Hopefully, you have understood these concepts by now and have practiced a little bit. But now, you will build some projects to implement the concepts.

By this time, you can add Express/Nest to your project as the Node.js framework.

e-commerce store

1. Storefront

   1. implement pagination and filtering.

   2. introduce OAuth using Google or Facebook Sign-in

   3. Add cart and checkout functionality with database transactions and ACID properties.

   4. Optimize the listing and get single product API queries.

   5. Write a rate-limiting middleware in your app to restrict API usage.

2. Store Management

   1. CRUDs for store owners to add products

   2. Implement RBAC for different roles, such as admin, read-only, etc.

   3. Save randomly fetched data in Redis and with caching strategies as per the use case.

   4. Provide support for a merchant to fetch or upload details in CSV files and implement a child thread in Node.js to perform some operations on the files.

3. Any e-commerce application use case you can think of.

4. Containerize your app and deploy on Render/Heroku.

Realtime Chat Application

1. Implement websockets for messaging

2. Add JWT or OAuth 2.0 for AuthN/AuthZ.

3. Save messages in Redis.

4. Use Redis for leaderboard analytics for any use case, such as someone who gives instant replies.

5. Containerize your app and deploy on Render/Heroku.

For monitoring and logging, I will suggest sticking with Sentry, as it’s a managed service with a free trial available.

Congratulations on reaching this point! You have learned and implemented essential concepts of backend development, from security to database to deployment.

Before moving ahead, I will suggest building the projects and APIs using the tech we have just discussed.

It’s time to introduce some advanced concepts and patterns.

Event-driven Approach

1. Overview of event-driven approach

2. Understanding of producers and consumers

3. Event Brokers and Queues

4. Architectural approaches

   1. Pub/Sub Model

   2. CQRS (Command Query Responsibility Segregation)

I suppose that you have been using a service-based approach while calling different sub-services.

Now, we will replace the synchronous approach with an event-based approach and make these modifications to your project.

Replace synchronous flows with an event-based approach in the projects

1. e-commerce

There are multiple services in your e-commerce project, such as:

1. cart

2. products

3. checkout

4. customers

to name a few.

Here, we will implement a basic consumer/producer pattern.

When a customer checks out, the events should be pushed to product and customer service to update the inventory and customer purchase history.

You can implement other use cases, such as payment notifications, abandoned carts, etc.

2. URL Shortener

You can implement analytics using the event-driven approach that whenever a URL is opened, the user performs some action or closes, you can trigger a notification and show it in analytics.

By this time, you have got your hands dirty on the surface of most of the things.

Let’s move to some advanced concepts. I am pretty sure you will be getting these used in actual jobs and production environments with real users.

Architectural Styles

1. Monolith

2. Modular Monolith

3. Microservices and its patterns

4. Serverless Architectures

Scalability

1. Horizontal vs vertical scaling

2. Loadbalancing: concept, algorithms and tools

3. Database

   1. Sharding

   2. Partitioning

   3. Replication and related use cases

4. CAP Theorem

Fault Tolerance and High Availability

1. Circuit Breaker Patterns

2. Choas Engineering

3. Auto-scaling principles

Misc.

1. gRPC, protobuf

2. Intrusion Detection & Prevention system with SEIM

3. Service Mesh

4. DDoS Mitigation Techniques

You can build the following projects to implement your learning.

1. Notification systems like Instagram and LinkedIn.

2. Real-time trading systems

3. Personalization/Recommendation Engines

4. AI-based chatbots with LLM

Once you have built these projects, you have learned enough of the backend concepts to think and build something on your own.

Although these will be some basic projects, they will help you to transition to backend projects easily.

When should I learn a new language?

Once you have mastered these concepts using Node.js and built projects, then you can transition to Python or preferably Golang.

Courses and tutorials I recommend:

1. NodeJS - The Complete Guide (MVC, REST APIs, GraphQL, Deno)

2. Understanding Node.js: Core Concepts

3. Fundamentals of Backend Engineering

4. Node.js Design Patterns

Cheers!

Welcome to the 30 new followers who have subscribed since last week.

We all know what Docker is and why it is used. As part of my Docker Internal Visualized series, we will learn how Docker works internally.

In the first part, we will explore how the Docker build process works internally, from image creation to layer creation.

1. Docker Build Command

docker build -t node-express .

This command creates a Docker image for a project using a Docker file.

Let’s break down this command.

1. docker build

Here, docker build informs the Docker client that you want to build an image.

2. -t 🏷️

-t refers to adding a tag to the image after it is created in the name:tag format. If no tag is specified, it will add the latest tag.

3. node-express

This refers to the name to be assigned to the image being built. It helps out later in identifying images by readable names.

4. “.”

The dot added at the end specifies the build context. When “.” is added, it refers to the current directory used as the build context.

.
├── index.ts
├── src/
├── Dockerfile
├── package.json
└── package-lock.json

Build context

The build context is just a set of files at a location that Docker will use to build. These files can be the project files against which the Docker daemon will create a build.

2. Docker client creating TAR file

After the docker build command gets entered, the Docker client comes into play.

It copies the build context files excluding those mentioned in .dockerignore and converts those into a TAR file.

The build time depends upon the size of the files inside the directory and it is essential to add unnecessary files in .dockerignore.

3. Docker Daemon (Dockerd) process TAR file

Docker client is a user-friendly interface to communicate with Docker Daemon via REST-API call.

When the docker build command is executed, under the hood, the Docker client sends the TAR file to Dockerd via an internal REST API call.

The Dockerd then unpacks, parses and loads the TAR file for executing each instruction one-by-one as per the Dockerfile and building the required Docker image.

4. Docker Filesystem and hash

Before moving further, it is important to understand how a docker image is internally built up.

Docker images are layers of files. Each layer represents a set of file changes. Each instruction in the Dockerfile such as FROM, WORKDIR, etc creates a layer after modifying the filesystem.

Each layer is created once and can be shared across multiple images.

Docker creates these layers because they are:

1. Immutable once created.

2. Reusable across different builds and efficient storage management.

Layers Stacking

1. FROM instruction creates a base layer and all other layers are built on top of it.

2. The subsequent instructions create and stack layers in the same order as defined in the Dockerfile.

Docker File Layer Hashing

When a new layer is created, Docker attaches a checksum SHA-256 hash. The hash is generated using metadata and file content internally by Docker itself.

It helps

1. uniquely identify layers,

2. securing from tempering because if some file changes the Docker will create a new hash and that image will not exploited.

3. verification

4. caching and improving latency and efficiency

We will discuss these concepts with examples ahead.

Note: In the animations, the base layer is on top and the following layers are below it, which is a visual representation mistake and the correct one will be updated in the next part of the series.

5. Dockerfile execution in Dockerd

1. FROM node:18-alpine

As discussed above, Docker Engine will first check the layered file on the local machine and if it doesn’t match any layer, it will

• pull an image from DockerHub

• create a new layer

• attach a SHA-256 hash to it.

2. Loading Build Context

The Docker will load the Build Context inside the execution context because the image will be created against these files.

3. WORKDIR /app

The WORKDIR command specifies the working directory to execute its subsequent instructions such as COPY, RUN, CMD, ADD, ENTRYPOINT, etc.

If the directory doesn’t exist, Docker will create this directory and all subsequent relative paths will be on top of this directory.

If this path is not specified, the Docker sets it at “/” by default.

The layer created at this step is very minimal and is empty and mainly modifies the metadata of the image. However, it is important to understand that the layer while not adding up to the storage, is effective for leveraging cached layers

4. COPY . .

This command will copy files from the current directory which are usually in the current directory to the previously defined WORKDIR, /app in this case.

With this command, all files in the build context are copied to the /app or whatever directory specified.

5. yarn install

It will simply run the command “yarn install” inside the /app directory where the project files are copied. It will also create a succeeding layer when the yarn install is successful.

6. CMD & EXPOSE

CMD instruction will set the command, you want to run when the container of this image will run. Similarly, EXPOSE exposes the port to connect with the container.

Here, the CMD and EXPOSE are empty layers similar to WORKDIR.

At this step our Docker image is ready.

6. Docker Image Layers

Let’s discuss Docker image layers.

To analyze Docker layers, you can run this command:

$ sudo docker history image-id

It will enlist all filesystem layers of the image.

As evident each layer represents a corresponding command in Dockerfile. Some add up to storage and others modify metadata.

Docker instructions such as WORKDIR, EXPOSE, and CMD, don’t add up to storage and modify the underlying image metadata.

All of the layers are stacked according to the instructions.

If a Dockerfile with the same instructions is used, the image will be created from cached layers including the base image pulled from Dockerhub.

In the next part of the series, we will explore Docker Caching and Invalidation in-depth with multiple use cases such as copy-on-write.

If you liked this post, please like, subscribe and restack.

Cheers.

While working with Node.js, you might have read about the terms blocking, non-blocking, and async.

These are part of the core foundational concepts of Node.js and refer to how Node.js, in practice, performs actions without blocking the main thread at scale.

In this article, we will start by exploring the terms blocking, non-blocking, and async, and later how they differentiate in Node.js.

As Node.js main process is single-threaded, these concepts are very important to understand I/O and concurrent operations.

Blocking I/O

In traditional I/O blocking programming, the I/O request will block the main thread execution until that request is completed.

In Node.js, Blocking refers to when JavaScript code can't be executed by the main process until a non-JS operation is completed.

It is important to understand here that the term blocking in Node refers to the execution of non-JS operations that will prevent JS code execution

Some blocking methods in Node.js are

1. Synchronous Libuv methods

2. Library’s methods, starting with sync

3. syncReadFile()

const fs = require('node:fs');
const data = fs.readFileSync('/file.md');

readFileSync() is a blocking method.

A CPU-intensive task can block the main thread too, but in Node.js terms, that is not referred to as "blocking." But that does indeed "block" the main thread

What happens when the main thread is blocked?

1. The JavaScript code inside the app can’t be executed. The JS code can be anything from async operations to timeouts, loops, etc.

2. As a result, the execution will come to a halt unless the sync operation is completed and unblocks the main process.

What operations can be done with the blocked main process?

Although the main thread is blocked, the OS will continue to perform its actions, such as queuing incoming calls, until the main thread is blocked, as visualized below.

Subscribe now

Non-Blocking I/O

The concept of non-blocking I/O is independent of Node.js, and we will discuss it generally and from a Node.js perspective.

Concept

1. Non-blocking I/O is simply an I/O operation that doesn’t block the main thread and waits for the I/O operation to complete.

2. Non-blocking refers to the behaviour of system calls that return immediately without waiting for the operation to complete. If the requested action cannot be completed immediately, the call will return an error or another indicator, rather than blocking the execution. The next operation in the call stack will execute.

3. It doesn’t block the main thread while waiting for I/O calls.

4. In traditional non-blocking systems, the caller usually polls to check the status of the non-blocking operation.

This concept is the inception of the event loop and event-driven architecture and supports concurrency, callbacks, and promises in a single thread.

Modern operating systems provide the underlying support required to build non-blocking operations. To make a system, implement the non-blocking approach; usually, system- and OS-level methods and APIs, such as file descriptors, epoll, etc., are used. Its implementation will be discussed in another post.

Approaches toward building non-blocking I/O

In traditional systems, a new thread is spawned when one is blocked and performs actions for other users. However, this approach is not an optimized solution.

1. Creating new threads each time is expensive as it consumes memory and is difficult to manage due to race conditions, sharing memory, etc.

   1. It’s not scalable, and as the number of concurrent users increases, the server can’t handle enough requests at a time.

   2. So creating multiple threads for multiple users is not scalable.

2. The solution here is to allow concurrency on a single thread.

   1. The OS provides an underlying architecture that implements an event loop to allow thousands of concurrent connections on a single thread. I will discuss this in-depth in another post in the upcoming weeks.

But now here arise two questions:

1. Aren’t child threads created in Node.js using child processes?

Node.js supports creating child threads, which are usually created when CPU-intensive tasks can’t be performed without blocking the main thread. To overcome this, child threads are initiated.

2. Node.js itself has a worker pool with four threads

It’s true. But those are entirely managed by the Libuv Library, which manages some blocking I/O operations away from the main thread and ensures the main process is always available to execute the app’s code.

Non-blocking in Node.js

"Non-blocking" refers to the behaviour of certain operations (typically I/O) that do not stop the execution of JavaScript code while waiting for something to complete.

Node.js follows a non-blocking I/O model powered by Libuv and Event Loop to enable this behaviour in Node.js

As opposed to blocking, non-blocking refers to the behaviour of such an operation which doesn’t block the main thread or execution of JavaScript.

Non-blocking usually refers specifically to I/O operations that do not block execution.

Although non-blocking is a language-agnostic phenomenon, a few built-in APIs, such as fs.readFile(), http.get() are called non-blocking APIs.

How does Node.js handle non-blocking I/O?

1. The Node.js Libuv engine manages non-blocking I/O operations by offloading them to workerpool threads.

2. When the worker pool thread is done, it will notify the main event loop and register its callback function.

   It is important to understand that the worker pool thread, unblocking the main thread, will be blocked until it finishes the I/O.

const fs = require('fs');

fs.readFile('/path/to/file', (err, data) => {
    if (err) throw err;
    console.log(data);
});

console.log('File read operation started');

This example above is one of the most quoted non-blocking APIs in Node.js. The flow of how it gets executed is visualized below.

Async

1. Async operations enable a program to initiate a task and then move on to other tasks before the first one is completed.

2. Async tasks are performed in a non-linear and non-sequential manner.

3. When the async task completes, it notifies the main program using mechanisms like promises or callbacks, but the under-the-hood task queue and event loop do the work.

The non-blocking paradigm enables support for async by building up an event loop that follows an event-driven approach, and the callbacks and promises perform the intended actions.

In Node.js, it’s managed through an event loop, which manages and controls the execution of all asynchronous callbacks and events.

Using both Async and Non-blocking I/O in one operation

In this visual of combining async and non-blocking operations, we visulized the flow. and their relationship.

1. As soon as readFile is executed, it’s offloaded to the Libuv workerpool thread. So, this is an example of a non-blocking I/O.

   1. In the Node.js world, this is now non-blocking.

   2. The Libuv thread executing readFile() is now blocked for other non-blocking I/O.

2. The console.log() on the very next line gets executed.

3. Similarly, both setTimeout() gets executed and are added to the Libuv timer.

4. While the non-blocking fs.readFile() is in progress, the first timer is reached, and its callback gets executed.

5. Meanwhile, the fs.readFile() operation is completed, and it notifies the main thread, and it’s callback gets registered to the task queue and later executed in the call stack.

6. And in the end, the last timer gets executed.

Here, it is evident that Node.js complements both async and non-blocking seamlessly.

How do you differentiate between non-blocking and async?

Both terms are used in Node.js, and they both contribute to making Node.js do I/Os and operations without making threads irresponsive.

1. Non-blocking specifically describes I/O operations that don’t block the main thread, while asynchronous is a general programming paradigm.

2. Non-blocking is typically implemented by Node.js runtime to perform I/O but async can be done via callback, promises, etc.

3. Non-blocking I/O can be implemented without async, like fs.readFile(), but without underlying support for non-blocking, async won’t work.

4. Non-blocking I/O responds immediately if data is available and status or error if it is not available. In async, it is not expected to respond immediately but performs actions as per callbacks and promises.

5. Continuous polling checks the status of non-blocking I/O, but that’s not the case when using async. However, Node.js doesn’t follow this approach of polling.

Benefits of using non-blocking & Async

1. It improves responsiveness as multiple tasks can be handled now.

2. Improved scalability.

3. Optimized resource usage and less CPU cycles wasted.

Conclusion

So, we learned the concepts of blocking, non-blocking, and async, with a special preference for non-blocking and async.

Thanks for reading.

Let’s connect on LinkedIn

Hello,

I couldn’t publish because I have been building Stripe’s Link-like service for my payment gateway, XPay, single-handedly. It was a great learning experience, from frontend to backend to security. I will be sharing a ton of content in the coming weeks.

In this blog, we will discuss why you should never save JWTs via client-side code.

If you have implemented JWT on your app, then you need to protect and store it safely on the client side.

If JWTs are not saved securely, they might be exposed to malicious actors.

Let's secure the JWTs and other sensitive data.

You need to send a cookie from the server-side header response instead of saving it on the client side with the following attributes:

Set-Cookie: accessToken=; Domain=abc.xyz.com; Path=/api/gateway; Secure; HttpOnly; Expires=Thu, 01 Jan 1970 00:00:00 GMT; SameSite=Strict


1. HTTP-only cookie: Always send JWTs via response cookies with an HTTP-only property. It prevents JS scripts from accessing the token, hence XSS attacks.

2. secure: It will ensure that cookies are sent over HTTPS only.

3. domain: by specifying the domain, you can ensure that the cookie can only be sent to that server.

4. SameSite: strict: A cookie that has this property set will prevent browsers from sending it in cross-site requests. Put simply, this implies that the request will not contain the cookie if the target site does not match the one that is now displayed in the browser's address bar.

5. path: To further restrict access to the JWT cookie within a server, the path can be specified.

6. expires: You can explicitly provide an expiration time, and the cookie will expire by then.

By implementing these measures, most of the client-side attacks will be mitigated, and your JWTs and other protected data to be stored on the client side will be protected.

Having said that, if there are other vulnerabilities on sibling servers or domains, handle cookies in an unsecured way.

In the next newsletter, we will discuss when JWTs are useful and when sessions are superior.

Stay tuned!

We are all used to installing NPM packages for React, Express, and other Node.js-based projects.

However, it is important to know how NPM secures the packages installed so that no tempered package can be installed in the codebase. I learned about these concepts when publishing internal npm packages for my company.

NPM uses these approaches:

1. Signing the package:
A package's validity is verified cryptographically when it is published to the NPM registry and signed with the author's private key.

To confirm that the package has not been tampered with during installation, npm checks the signature against the author's public key.

2. Integrity hash in package-lock.json file:

Each package has its own "integrity" field in the "package-lock.json" file. The package's contents, including any required dependencies, their respective versions, and the SHA-512 hash, are hashed and stored in this field.

Each time you use the "npm install" command to update your project, npm will check the integrity hash of each package and its dependencies against the one saved in the "package-lock.json" file.

If the hashes are the same, the package has not been tampered with, and the installation can continue. If the hashes don't match, npm won't install the package since it thinks it's fake.

3. Scanning for vulnerabilities:

NPM's "npm audit" tool examines the currently installed packages for known security flaws and suggests solutions. This aids in making sure the packages being used have no known security flaws.

There are other recommended approaches, but these three are widely used.

Thanks for reading.

Subscribe now

1. Don’t write only code

If you are just focused on writing endless lines of code without giving a thought to a few things, you are setting yourself on a path of coding monkey and won’t reach higher positions such as lead, architect, staff engineer, etc.

1. Software engineering is more than just writing code.

2. When you get good experience in a specific domain, you need to start mentoring the juniors.

3. You need to start leading the projects or features related to that domain.

It will elevate your worth in the company and help you reach higher positions.

2. Clean Code Practices

This is often overlooked by software engineers, especially at junior levels, who are more eager to ship code, usually at early-stage startups.

Velocity is indeed required, but if done at the cost of coding practices and scalability, it will impede progress in the long term.

As a software engineer, you need to explain to your manager the impact of not hurrying and instead taking time to write the best code to make sure nothing crashes in production.

3. Following Test-Driven-Development

Often, TDD is missed at the cost of ensuring the stability of the codebase for faster shipping of the features.

Though it may look like you are delivering faster, in fact, you're just one step away from a surprise production crash.

If the code follows proper unit and integration testing and is written following TDD principles, it will stabilize the app; no surprise changes will break your app.

It’s better to test yourself than customers doing it.

4. Understand Performance Optimization

Irrespective of your tech stack, you need to master the principles of optimizing application performance.

To start with coding, there are generally established, optimized, and faster practices and methods in all programming languages. So, you need to follow that path. Most of the time, the fancy solution you use is not required.

Let’s say you are a front-end developer. You need to learn about

1. Web Vitals,

2. how the browser works and renders pages,

3. What are the best practices to secure things at the front end?

4. How to serve static assets via CDN?

5. What is the best way to deploy the app?

6. How JavaScript thread execution takes place

Similarly, as a backend developer, you need to learn

1. Correct status codes and messages

2. Understanding the databses.

3. Caching and reducing latency of APIs

4. Decreasing the number of round APIs

5. Understanding the networking side of APIs

6. Load balancing, etc

Learning and implementing these concepts will upgrade your game as a software engineer.

5. Don’t follow the approach of one programming language to another

I have often seen Python developers just using let everywhere without giving a thought to whether that is actually required or "const" can be used as well. To me, it indicates a junior-level developer. As a pragmatic developer, you should learn syntax, why it should be used, and do accordingly to write better code.

6. Learn about the architecture and system designs of the established products

Software engineering is always changing, and big companies are always putting in and experimenting with new tech. You need to follow and learn from them. It will upgrade your level as a solution-oriented engineer who can help solve problems.

These are some of the pieces of advice that I can give to someone willing to thrive at the job.

What is a source map?

A source map is a file that maps a file’s original source code (such as a JavaScript file) to its generated version (such as code that has been compiled and minified).

The source map file has the names of the files, the line numbers, and a map of the code that has been compiled from the original source code. By displaying the original source code, the source map assists developers in troubleshooting built programs.

Web developers use source mapping to translate, shrink, and bundle JavaScript code to make websites run faster, but this makes it harder to find bugs. Source mapping can help debug and create CSS, graphics, and other programs.

Is a source map necessary?

Source maps are important because they let developers troubleshoot their original source code while working with the generated version. This makes the code easier to read and understand.

Source maps let you connect the original source code to the code that was made, which solves the problem. When a developer is debugging compiled code, they can use the source map to view the original source code, which makes it easier to identify and resolve errors.

Source maps also let developers debug code on a live website or app. They can use the browser's developer tools to look at the original source code and set breakpoints, just like they would when developing locally.

Source map and security implications

A source map is obviously helpful for debugging, but it also raises security concerns. It can expose a program’s or website’s original source code, which may reveal things like passwords or private data.

1. The information included in a file’s source code, such as passwords or other secrets, could be exposed if a source map were to be used to read the file.

2. An attacker who is hunting for vulnerabilities may find source maps helpful because they show the structure and organization of the software.

3. A misconfigured source map could expose the original source code to attackers or harmful third parties.

4. When an attacker obtains a source map, they can use it to reconstruct the original programs, which could lead to the discovery of security flaws or other private details.

5. With the help of source maps, attackers can insert malicious code into legitimate files, potentially granting them execution privileges on the victim’s computer.

   Thanks for reading Zero To Architect! Subscribe for free to receive new posts and support my work.

How to not generate a source map in production?

The process for avoiding the creation of a source map during the building and compiling of your code will vary depending on the tool or framework you employ. Some instances are as follows:

// Webpack
module.exports = {
  devtool: 'none',
  // ...
};

// Babel
babel --sourceMaps false script.js

//typescript
{
    "compilerOptions": {
        "sourceMap": false,
        //...
    },
    //...
}

//Rollup
export default {
  input: 'src/index.js',
  output: {
    file: 'dist/bundle.js',
    format: 'cjs',
    sourcemap: false
  }
};

How to generate a source map for a React application?

1. Set “GENERATE SOURCEMAP=false” in your .env file in the root of your project directory.

2. Change the React scripts command

// package.json
"build": "GENERATE_SOURCEMAP=false react-scripts build"

3. You can use the webpack configuration as discussed above.

Is there a way to save source maps securely and access them?

Source maps can be protected against unauthorized access and disclosure of important data by being stored in external, password-protected files.

Following these instructions will allow you to securely store your source maps in external files:

1. Take advantage of a build tool that gives you control over where the source map file is written. The dev tool option in Webpack, for instance, lets you set where the source map file is written.

2. Protect the original map file by uploading it to a file hosting service that offers encryption. The source map file, for instance, can have its rights restricted using AWS S3 so that only certain people can access it.

3. Use an authorization tool to limit who can access the source map file, such as JSON Web Tokens (JWT).

4. Utilize Content Security Policy (CSP): CSP is a security feature that aids in detecting and mitigating specific attacks, such as Cross-Site Scripting (XSS) and data injection attacks. It can be used to restrict who can access the source map file by specifying which sources are authorized to load content in the browser.

It’s worth noting that these measures can aid in securing the source map file, but cannot ensure the entire security of the source code. Reviewing the codebase and expunging private details is a recommended best practice.

How to debug your code without a source map?

1. Pretty-print the code: Some minifiers and transpilers include white spaces and comments to make the code more legible. This simplifies code comprehension and debugging.

2. Set breakpoints and step through code with a debugger: Many recent browsers feature built-in developer tools. The debugger can help you comprehend the code flow and find bugs even if it’s not as readable as the original source code.

3. Use a decompiler: A decompiler can turn compiled code into readable code. The resulting code may be simpler to comprehend and troubleshoot.

4. Knowing the code structure may help you understand it without a source map. Even if the code is minified, you may be able to discover and debug a method that handles a feature.

5. Logging: Use console.log commands to output variable values to understand the code flow and discover faults.

Thanks for reading.

Every developer strives to write the best code possible. It ensures that code is readable, maintainable, scalable, and consistent. Usually, developers follow best practices, industry standards, unit testing, manual PR reviews, etc.

However, code quality and security can drastically improve by incorporating static code analysis into your projects and a code base that performs code quality checks automatically.

What is static code analysis?

In plain English, static code analysis refers to analyzing the code before it’s running to see if it meets certain standards and rules defined.

How does static code analysis help improve code quality?

1. Making code less complex: A case of cognitive complexity

   Who likes code that takes hours to understand because it comprises hundreds of lines and all business logic is put inside a single function? More complex code might make it more prone to bugs and less scalable

   .

   The complexity of software is measured by the most popular method, cyclomatic complexity. It counts the number of distinct routes that can be taken through a piece of code, assisting programmers in determining how intricate and potentially error-prone their software might be. A more complex and difficult-to-maintain codebase is often indicated by a higher cyclomatic complexity.

   

   A static analysis tool such as SonarLint or ESLint will provide cognitive complexity to code to ensure it is readable and divided into smaller chunks.

2. Formatting and Styling Code

   Every software developer has a personal preference for formatting and styling conventions. Some like single quotes, others double, etc. Apply this to a project with tens of developers, and this will create a complete mess.

   Even if developers are asked to apply the conventions, they can’t be expected to remember all the rules. All programming languages provide linting tools to adhere to conventions.

3. Type Checking

   Let’s assume a use case where a function expected a boolean, but that boolean was passed wrapped in a string. We know the app will break for sure.

   The type checking rescues here. In the case of JavaScript, Flow, and TypeScript can ensure that the written code is actually predictable.

   It ensures that all the parameters passed and their children, if any, are all defined at the level of the actual values as well.

4. Detecting Bugs and Errors

   Let’s say you add a duplicate variable to the code, and the code is shipped to production. Though the issue can be found in the unit tests, it can be caught earlier by using static analysis tools. Similarly, redundant and unused code can be timely identified by analyzing the code.

5. Security issues

   Often, bad coding practices introduce vulnerabilities into the codebase and the application, which can be exploited later via XSS, privilege escalation, etc.

How do you implement static code analysis in your project?

1. IDE: First line of defense

   Almost all linting tools can be installed in the IDE, ensuring that all standards are followed while you code.

2. Git Pre-hooks

   Run all lint tools before pushing the code to the remote repo by integrating with pre-hooks tools such as Husky to ensure that all standards are being followed.

   Tools such as ESLint also provide methods such as —fix to fix most of the issues found by ESLint.

3. CI Pipelines

   The static analysis tools can be run just after a PR is raised or before pushing the code to production. It should fail the pipeline, PR, and push to the production environment if it fails.

Tools for static code analysis

1. ESLint

   The most widely used lint tool for JavaScript maintains conventions across the codebase and captures inconsistent formatting and styling. It comes with predefined rules, which can be modified.

   Pylint

2. Prettier

   Prettier is the most popular formatting and styling tool to ensure consistency and uniformity across the codebase.

   

3. SonarLint, SonarCloud and SonarQube

   SonarCloud is a popular cloud-based code analysis tool that can be integrated with the GitHub repo. SonarQube, a self-hosted tool, can be added to your pipelines.

   Similarly, SonarLint can be added to the IDE and provide the same configuration and enforce rules directly in your IDE.

4. LGTM

   GitHub provides a tool called LGTM to directly inspect the codebase via GitHub Actions. Further details can be found on the Github LGTM

   Some other best and most popular tools are:

5. DeepSource

6. DeepScan

7. Codacy

8. JSHint

9. JSLint

10. Dependabot

11. npm-audit

12. pytype

13. pylint

Thanks for reading.

Over the last one and a half years, I have been working as an architect, and most of my time is spent on making decisions and building scalable solutions. Besides domain-related hard skills, the working knowledge of DevOps and AWS helps me out while creating the best solutions.

I am convinced that every developer should know about DevOps and at least manage the Ops related to his domain.

In this article, we will learn about what DevOps is, what the different parts of it are, what you should learn, and how it will help you further in your career. Even the basics of DevOps are also required in the job description nowadays.

What is DevOps?

DevOps is a set of practices and cultural beliefs that emphasize collaboration and communication between software developers and IT operations professionals. Its goal is to automate the software delivery process and increase the speed, quality, and dependability of software releases.

DevOps integrates software development, testing, deployment, and operations into a single continuous process, allowing for faster and more efficient software development and deployment. It also includes the use of tools and technologies that automate the various stages of the software delivery process, such as version control, continuous integration and deployment, monitoring, and infrastructure as code.

DevOps technologies and concepts you must know

1. Docker

Docker is a containerization platform that lets developers deploy their apps consistently and portablely.

Docker lets developers create container images with all the software dependencies, libraries, and configuration files needed for their applications. Users can share these images. To ensure software consistency across development, testing, and production environments, these images can be shared and used.

A Docker client tool interacts with the Docker daemon, which manages images and containers. Docker’s command-line and graphical interfaces can manage and deploy containers.

2. Kubernetes

Kubernetes (K8s), an open-source container orchestration platform, automates containerized application deployment, scaling, and management.

Developers can deploy their apps in lightweight, portable containers with Kubernetes. Kubernetes manages and scales these containers efficiently across a cluster of nodes.

Kubernetes uses a master and multiple worker nodes. The cluster’s master node schedules worker node containers. Workers run containers and receive updates and instructions from the master node.

Kubernetes provides:

1. Automatic scaling: Kubernetes scales containers based on demand to ensure applications have enough resources.

2. Self-healing: Kubernetes can detect and replace unhealthy or unresponsive containers, ensuring application availability.

3. Load balancing: Kubernetes distributes traffic across containers for responsive applications.

4. Controlled application update rollouts and rollbacks reduce downtime and error risk with Kubernetes.

5. Storage orchestration: Kubernetes manages container storage to give applications access to data.

3. Infrastructure as code (IaC)

Infrastructure as code (IaC) uses code to automate and manage infrastructure. Instead of manually configuring servers, networks, and databases using a GUI or CLI, it defines them in declarative code.

IaC benefits include:

1. Consistency: IaC consistently provisioned and configured infrastructure, reducing human errors and misconfigurations.

2. Reusability: IaC code can be shared across teams and environments, making infrastructure provisioning faster and easier.

3. Scalability: IaC quickly and easily scales infrastructure to meet application and user needs.

4. Version control: Developers and infrastructure teams can collaborate on IaC code using version control systems.

5. Auditability: IaC tracks infrastructure changes, making troubleshooting and debugging easier.

Terraform and CloudFormation allow declarative code descriptions of infrastructure components that can be versioned, tested, and deployed automatically. Dependency management, variable support, and modularity simplify infrastructure management.

4. Continuous Integration/Continuous Delivery (CI/CD) Pipelines

Continuous Integration/Continuous Delivery (CI/CD) allows developers to automate software building, testing, and deployment to production.

CI/CD pipelines automate code building, testing, and deployment. Developers start the pipeline by committing changes to version control.

The pipeline then executes:

1. Code creation: The pipeline compiles code from the version control system.

2. Testing the code: The pipeline runs a suite of tests to check for bugs and regressions after code changes, such as unit, integration, and functional tests.

3. The pipeline packages the code and dependencies into a deployable artifact like a container image or ZIP file.

4. The pipeline deploys code changes to a staging environment for testing before deploying to production.

5. Monitoring code: The pipeline checks the staging application for errors.

6. The pipeline promotes code changes to production if the staging environment is stable and error-free.

7. CI/CD pipelines speed up and improve software delivery to production.

Automating the building, testing, and deployment processes reduces manual errors and downtime and improves visibility and control over the software delivery process.

5. Deployment Strategies

Software modifications are released and deployed to production environments using deployment strategies. Speed, risk, and complexity vary with deployment strategy. Deployment methods include:

1. Blue-Green Deployment: This technique maintains two identical environments for production and staging. The staging environment hosts the new application, which is tested and certified before traffic is moved to it.

2. Rolling Deployment: This technique involves releasing changes to a small group of production servers, checking that they function, and then gradually rolling out the changes to the other servers.

3. Canary Deployment: This method deploys changes to a limited group of users or servers in the production environment while keeping the rest of the traffic on the old version. After validation, traffic is gradually switched to the updated version.

4. A/B Testing: This method involves deploying two versions of the application, A and B, and routing a percentage of traffic to each version. Both versions are compared, and the superior one is maintained.

5. Immutable Deployment: Instead of upgrading the environment, each deployment creates a new one. This makes the environment consistent and trustworthy and decreases the possibility of errors caused by changes between environment versions.

Each deployment approach has pros and cons, and the ideal one depends on the organization’s needs. The application’s size and complexity, risk tolerance, resources, and deployment speed should be considered while choosing a deployment method.

6. Site Reliability Engineering

Site Reliability Engineering (SRE) integrates software development and operations to construct and run reliable, scalable, and efficient software systems. SRE teams monitor system uptime, latency, and error rates.

SRE relies on numerous fundamentals:

1. SRE teams monitor system performance, including uptime, latency, and error rates. This data helps uncover issues and inform system improvements.

2. Automate everything: SRE teams automate to eliminate human error and boost efficiency and consistency. Automated testing, deployment, and issue response.

3. SLOs and SLAs: SRE teams set SLOs and SLAs to guarantee system performance meets user needs.

4. Perform blameless post-mortems: SRE teams investigate incidents to determine what happened, why, and how to prevent them from happening again. Instead of blaming, learn from mistakes and improve the system.

5. SRE teams experiment to test new ideas and improve the system. A/B testing, canary deployments, and other methods are used to test changes before implementing them.

6. SRE teams collaborate with development, testing, and operations to ensure system reliability and efficiency. Sharing information, resources, and best practices.

7. SRE is a mentality and practices for designing and running reliable, scalable, and efficient software systems. SRE teams can increase system reliability, performance, and user experience by following these guidelines.

7. Cloud Services

Cloud services are internet-delivered services from cloud computing providers. These services offer computation, software, and storage. Cloud services are usually subscription-based and accessed via a browser or client application.

The most popular cloud service is AWS. Its most useful services are EC2, S3, Lambda, RDS, Secret Manager, and more.

How will learning these DevOps concepts and tools help you as a developer?

1. A better technical solution

A developer can create more resilient, scalable, and solid technical solutions. Knowing how different parts of technology work can help in making better decisions. For example, the decision to use Lambda vs. deploy a service on EC2 with respect to speed and cold starts can only be made when you have gotten to know in and out of these technologies. And this can only be possible if you have some background in DevOps and AWS.

2. Production Debugging

DevOps can help developers debug production difficulties. Debug production issues using DevOps principles with these steps:

1. Monitor production difficulties with monitoring tools. Monitoring can help you catch problems early. DevOps monitoring is automated and combines with other technologies to give real-time application visibility.

2. Reproduce: Use your DevOps process to replicate an issue in a development or testing environment. Reproducing the problem helps diagnose and solve it.

3. Collaborate with operations teams to understand the production environment and identify configuration or infrastructure issues that may be causing the problem. Use DevOps concepts to collaborate and communicate with operational teams to fix the issue.

4. Fix: Use your DevOps process to push the fix to production after finding the issue. Continuous delivery and automated testing can assist in deploying the patch quickly and safely.

5. Monitor again: After delivering the fix, monitor the production environment again to confirm the issue is repaired and no new issues arise. This verifies your repair and production stability.

Developers can reduce downtime and improve the user experience by debugging production issues using DevOps techniques.

3. A better approach toward development

Developers can improve their development methods with DevOps.

1. Agile Development: DevOps encourages rapid iteration, collaboration, and continuous improvement. Developers can respond to changing needs, provide features faster, and get stakeholder input by working in Agile.

2. Collaboration: DevOps encourages collaboration between developers, operations, and other software development stakeholders. Developers can better understand the production environment and design deployable, operable software by engaging with operations teams.

3. DevOps emphasizes monitoring and feedback throughout development. Developers can enhance their code by analyzing production application performance and availability.

Conclusion

To summarize, it can be said that DevOps is a necessary tool for developers to learn, as it will help them in debugging, developing better solutions, and developing an improved skill set to match market demand.

Developers want their APIs to be reliable and resilient.

But there is an area that is often overlooked: are your APIs predictable and retriable?

Let’s explore some examples.

Customers getting double charged: A classic example

A customer is using your payment APIs, and as soon as the process is initiated, there is a network issue on the customer’s end, and he is unable to receive a response. As soon as the network is back, they click on the pay button again. What should happen now?

If the payment process is restarted, the customer will be double-charged. Which will dent the credibility of the payment processor.

Order and Inventory Management Systems

Similarly, let’s say you have an e-commerce OMS app. The store owners process some orders but, for some reason, again perform the same actions. The system should not reprocess but instead, return the original response.

These problems can be resolved by introducing idempotency in the APIs.

What is idempotency?

Idempotency is a mathematical and computer science concept that refers to the quality of some operations or functions in which performing them several times has the same effect as performing them once.

An idempotent operation can be repeated an infinite number of times and still provide the same outcome.

Idempotent APIs enable API users to retry calls in the event of a network outage or other problems without having to worry about producing duplicate rows in a database or charging the client several times.

HTTP Methods ensuring idempotency

HTTP methods such as PUT, DELETE, and GET are idempotent in nature.

1. GET: Repeating a GET request will return the same resource representation without changing the resource on the server.

2. HEAD: The HEAD method, like GET, obtains metadata about a resource but not the resource itself. It can be used to check for the existence of a resource, get headers, or find the last change date without modifying the resource itself.

3. PUT: If the resource at the specified URL already exists, a PUT request will replace it with the new representation sent in the request body. PUT will build the resource if it does not already exist. PUT is idempotent in the sense that repeating the same PUT request with the same data does not affect the resource in any way other than the initial request.

4. DELETE: When you issue a DELETE request, the server deletes the resource if it exists, and it simply provides a successful answer if it does not. Multiple DELETE operations on the same resource will not have different outcomes beyond the first deletion, making it an idempotent operation.

How can a developer build idempotent APIs?

A developer can add idempotency in various ways.

1. Idempotency Keys

Idempotency keys are the building blocks of idempotent keys. A key, preferably a UUID, is used to identify a request. Whenever the API is called again with the same key, it will return the original response.

The keys are either external or internal.

• External Idempotency Keys: If the API is integrated into 3rd-party apps, then that system can pass the key to be identified as the idempotency key.

• Internal Idempotency Keys: If the system generates a key against each request and attaches that on the basis of an ID, then that key will be called an idempotent key.

Saving Idempotency Keys

The idempotency keys are usually short-lived and can be reused with new requests after a specified time.

1. Redis: Redis or other caching solutions can be used to save short-lived idempotency keys.

2. TTL indexing: Time-to-live indexes are often used to save a value in a database to be deleted later after a given time interval.

2. Using HTTP Methods

Using HTTP methods such as GET, PUT, and DELETE will ensure that the API resources are by default idempotent.

Let’s write some code

Here's an example of how to utilize the Express.js framework to create an idempotent API endpoint in Node.js, including the use of an idempotency key to ensure that repeated requests do not have unintended consequences.

• For upgrading a user's email address, we establish an Express.js application and configure a POST request endpoint (/users/:userId/email).

• To uniquely identify each request, we develop and employ an Idempotency-Key header. If you use the same idempotency key in many requests, the server will recognize it as a duplicate and reply accordingly.

• The server determines whether the idempotency key has previously been processed. If it has, the server responds with a response indicating that it has already handled the request.

• To prevent duplicate processing of the same key, processed idempotency keys are placed in a Set.

Final Thoughts

It is critical to consider the probability of failure in a distributed system and how to address it while developing APIs that are both resilient and predictable. Retry logic on clients and server idempotency are two strategies that can help achieve this goal and are reasonably easy to implement in any technological stack.

Here are a few fundamental principles to remember while building clients and APIs:

1. Ensure that failures are dealt with consistently. Request that clients retry activities on remote services. Failure to do so may result in inconsistent data, which may cause problems in the future.

2. Make certain that failures are handled in a safe manner. Allow clients to pass a unique value and retry requests as needed by using idempotency and idempotency keys.

Productivity is contextual and subjective. It can’t be generalized.

I manage and lead 8 to 10 engineers on average across different teams and use different measures to determine their performance and productivity instead of one rule that fits all.

Let’s start.

1. Year of experience in a specific tech stack

2. Difficulty level of new tech stack for a new task

3. Relevance of tasks with the tech stack

4. Ability to learn any new skill to fulfill the job

5. Knowledge of relevant tech ecosystems

Example 1: Experience and task in the same tech stack

Let’s say there is a developer with three years of experience in JavaScript and React. Xe is assigned a task of five-story points, which is relatively fair and expected to be completed if no blockers occur.

If the task is completed within the timeframe, then the developer is performing well.

But if the random tasks are taking more time, then there needs to be a discussion.

What is to be discussed here?

As a lead or manager, I will first sit down and evaluate what went wrong here. There are chances that

1. There was tech debt that needed to be cleared first.

   1. But who created that in the first place? Was it some other developer?

2. The feature affected other parts of the code and needed to be handled, and it took more time than anticipated earlier.

   1. Was the codebase not scalable?

   2. How can we ensure that it doesn’t happen in the next feature implementation?

3. The person took more time.

The first two options clearly illustrate that the developer might not be wrong and could be given leverage. But in the last one, the performance is quite low.

Example 2: A developer is assigned a task of an overlapping tech stack

A backend developer with an Express.js and Node.js tech stack can be expected to take over Nest.js with ease, and his task estimates can be extended from 10% to 25%. However, if the same engineer takes double the time, then it means either he lacks basic JS OOP skills or is a slow learner.

The latter is acceptable, but the former is not and will directly fall under poor performance, and the developer should be considered less productive.

Example 3: You are given a task above your skill level

Assume you have average database skills and know the syntax and APIs to work with. Your next task is implementing sharding and partitioning with eventual consistency and maintaining connection pooling.

It’s a difficult task that requires thorough knowledge of the internals of the database.

What to do as an engineer

1. First, discuss with your manager that you don’t have the current skillset for this task, but you will learn it and finish it.

2. Discuss the timeline. If a person with the required skillset needs two weeks, then it should be four for you.

3. Clear expectations earlier to avoid any confusion and delivery delays later.

4. This will ensure that, as a developer, your performance level is as per your skillset, with room to improve.

What to do as an engineering lead and manager

1. Understand the situation and skill level of your resources.

2. Set expectations for the deliverable accordingly.

3. Even if the candidate is able to deliver 50–70%, it means xe performed well and couldn’t be deemed a non-productive engineer.

Tangible criteria to judge an engineer

Irrespective of any task, the developers must be following these practices:

1. Clean code practices being followed

2. Unit and integration testing

3. Scalable code

4. Number of bugs and issues reported

5. Git standards being followed

With different use cases, I have tried to establish that it is not that easy to evaluate even one developer under different scenarios.

So, interact more with your team and ensure that you understand their side of the story before evaluating performance and productivity.

Thanks for reading.

McKinsey published a report and laid down some productivity indicators to build a framework for tech companies.

As soon as I read this report, the first question I asked myself was, Do I measure the productivity of my team members? If so, do I use one generalized approach for all? The answer is no.

Is it important to measure the productivity of software developers?

Yes, every company needs the best talent and wants to replace the least productive engineers with the most productive ones. However, the productivity parameters of one developer completely differ from those of the other.

It is impossible to apply one formula that fits all and expect accurate results. It will be disruptive.

Let’s say there are three developers, A, B, C, and D, working in a team and putting in 100% effort.

Developer A is working on new features and delivering them regularly.

Developer B is reducing tech debt by optimizing code for scalability and paving the way for new feature development at a rapid pace.

Developer C is blocked most of the time due to third-party dependencies and couldn’t get it delivered.

Developer D is building with a completely new stack, and the development pace is slow, but things are getting done as per contractual expectations.

Who will get more recognition and be considered more productive in typical C-level meetings? Does it mean, by any standard, that one is less productive than the others? A person solving bugs or reducing tech debt might seem less productive than one delivering features in a typical tech company.

A summary of the McKinsey report

McKinsey thinks they have a solution to measure the developer’s productivity.

This article explores the significance of measuring software developer productivity and proposes a novel method for doing so. It emphasizes that, in contrast to other corporate operations, software development has generally been undermeasured. However, because most businesses now rely substantially on software, it is critical to analyze and optimize developer performance.

Given the complexity and collaboration required in software development, the essay acknowledges the difficulties in assessing developer productivity.

The article’s innovative technique involves assessing production at three levels: system, team, and person, using DORA and SPACE metrics and Opportunity-focused metrics.

It introduces new measures, including deployment frequency, change lead time, and contribution analysis, to evaluate performance and find areas for improvement.

The essay emphasizes the need to avoid measurements that are unnecessarily simplistic, such as lines of code, and encourages organizations to move beyond antiquated mindsets that imply software engineering cannot be measured.

The article concludes with suggestions for getting started with measuring developer productivity, such as learning about software development, assessing existing systems, developing a plan, and understanding that measuring productivity is contextual and aimed at improving the entire development environment. The goal is to boost developer productivity and overall corporate performance by increasing openness and insight.

An analysis from a developer’s and engineering lead’s perspective

The report is more focused on building a framework for C-level managers than considering the software development lifecycle and the developer’s journey from building to delivery.

1. A non-engineering person gauging developers' productivity and influencing decisions

This is where the problem starts.

It’s just a button or an API connected to the database.

To a non-techie, it might look like just a feature, but the concepts of tech debt, architectural decisions, and feature implementation are naive.

The question is, when the CTO is spearheading the technical side, shouldn’t he be calling the shots and firing and hiring developers based on their performance?

Not to mention non-engineering people, I have witnessed leads and managers struggling with developers working on the other tech stack. I have witnessed backend leads who don’t understand how a page loads on browsers and, out of ignorance, rank frontend developers low.

Learn the basics. All C-suite leaders who are not engineers or who have been in management for a long time will need a primer on the software development process and how it is evolving.

It’s a welcoming statement, but not all tech companies are building the same product. R&D, architectural decisions, and overhead vary. The developer and software lifecycles in a fintech startup are different from those in a typical SAAS. So the question is: are C-level managers willing to learn from the context of their product?

2. Entire focus on coding

From both a productivity and personal-experience standpoint, maximizing the amount of time developers spend in the inner loop is desirable: building products directly generates value and is what most developers are excited to do.

The document thinks that the more code a developer writes, the more productive he or she is. The irony is that it indirectly insists on building a coder mindset in organizations.

There is no mention of how much different aspects of software engineering, such as building a system design, taking architectural decisions, and discussing scalability, influence the course of the product.

3. The outer loop is completely generalized for all developers

The absence of platform engineering and DevOps standardization will impede development progress and make developers focus less on their jobs.

It makes sense that the developers should be unblocked for CI/CD pipelines and deployments, but excluding meetings, security, and compliance from a developer’s activity is rather a generalization.

It’s true for most tech companies, where unnecessary prolonged meetings and standups happen and should be reduced to a point where developers just attend meetings where they are actually required.

A developer working in Fintech and Banking is considerate of security and compliance. If development evolves the product, then obviously these factors come into play. Who will then implement those features?

4. No mention of automated and manual QAs or DevOps engineers

Productivity can vary from developer to developer, depending on the job description.

Developers are not the only ones building the product and need to be productive. The QA team and DevOps play equal roles. Here, no indicator is defined to measure their productivity.

The manual QAs and the ones doing automation need different criteria, as do the DevOpsers.

5. Developer Velocity Index

DV, in fact, adds to the efficiency of a developer’s and the overall software development cycle. The developer’s productivity will fall to zero if they get stuck in CI/CD pipelines, manual unit testing, tedious code promotion strategies, etc.

6. Overlooking non-coding contributions of developers

For example, one company found that its most talented developers were spending excessive time on noncoding activities such as design sessions or managing interdependencies across teams. In response, the company changed its operating model and clarified roles and responsibilities to enable those highest-value developers to do what they do best: code.

The company went wrong here, and this model is not a viable one.

It can be assumed that the most talented developers should know the ins and outs of the product and the internal dependencies overlapping across the teams. Who is supposed to take the lead, create, or assess designs and system designs? Are those talented developers or others hired for these specific roles? Were those developers completely turned into coding bots?

Let’s say there is a major feature implementation that requires ongoing communication between the teams. It’s a common practice that managers, with the help of developers, handle this and point out the shortcomings and successes of the integrations and implementations.

Creating a separate role for this creates only friction rather than good code and products.

7. The framework doesn’t address contextuality

Remember that measuring productivity is contextual. The point is to look at an entire system and understand how it can work better by improving the development environment at the system, team, or individual level.

It’s a welcome point in the report, and it’s true.

At my job, my manager gauges my productivity and performance in a different context than I do for my team members on a different tech stack.

At the team level, my developers get blocked for a couple of days due to an external issue, although a feature has been completed. When they get unblocked, they might have to change a feature implementation due to a third-party requirement, and the delivery gets delayed. In this context, the team is not a non-productive one. But if it is clearly monitored, the team is indeed productive, as they have delivered the feature in the time required to do all the work.

To some C-level managers, the developer might look non-productive as the feature is getting delayed, but the reason is not the productivity issue.

Similarly, if the team or developer underestimates the ETA and there are no blockers, they fail to defend against delays, and productivity and performance issues will come into the spotlight.

It’s easier to gauge performance with a given set of points, such as clean code, unit testing, number of bugs, tech debt accrued, etc.

8. The impact of the developer on the team is greater than the coding

This is one of the most important aspects of an outstanding developer, but there seems to be no mention of it. If you are a senior developer on the team, you will be expected to help out the juniors, review code, and suggest solutions. The amount of time spent coding will not be a big factor in your assessment.

The greater your impact, the more productive and exceptional you will be considered.

Next Sunday, I will share how, as a developer or engineering lead, you can assess your and your team’s productivity contextually, unlike the generalized one. You can subscribe to get that blog directly in your inbox.

Summary

It’s a very welcoming report, but it comes with some basic flaws. Every manager wants to know the productivity level of his or her team and how to improve it. But it can’t be generalized and applied across organizations. Every developer or team has different tasks and circumstances. Similarly, each developer comes with different strengths and weaknesses, which have an impact on the product being built. Some are good at unblocking teammates and building solutions, while others write exceptional and optimized code. Those who implement this framework will produce more coders than software engineers who build solutions.

Thanks for reading.