Today I Learned

Partition for manageability, Sharding for scalability.

Partitioning
Dividing data into segments (partitions) for easier management or to group related data together.
Often used within the same system and transparent to the application.

Sharding
Splitting data across multiple databases or servers to distribute the load and scale horizontally.

Each shard operates independently, and often, the application needs logic to direct queries to the correct shard unless the data storage system transparently supports the redirect.

CAP Theorem

System Design

The CAP Theorem is a principle that says a distributed database can only guarantee two of the following three things at the same time:

Consistency (C): Every read gets the most recent write.
Availability (A): Every request gets a response (either success or failure).
Partition Tolerance (P): The system keeps working even if parts of the network fail (e.g., servers can't talk to each other).

Example:

Imagine you have a system with multiple servers that store user data.

Consistency (C): If you update your email address on one server, all other servers will immediately reflect that change.
Availability (A): Even if one server goes down, the system will still respond to your requests.
Partition Tolerance (P): Even if a network issue occurs and some servers can't communicate with others, the system will continue working.

The CAP Theorem says: You can pick two of the three properties (Consistency, Availability, Partition Tolerance), but not all three at once. For example:

CA (Consistency + Availability): If you want consistency and availability, the system might fail when the network partitions.
CP (Consistency + Partition Tolerance): If you want consistency and partition tolerance, the system might not be available (i.e., not respond to requests).
AP (Availability + Partition Tolerance): If you want availability and partition tolerance, the system might serve stale data (not consistent).

Why You Can't Have All Three

1. Consistency and Availability without Partition Tolerance (CA):

Scenario: Imagine a system where you have two servers. You want to guarantee that every request gets a response (Availability) and that all users see the same data (Consistency).
Problem: If a network partition occurs (e.g., one server can't communicate with the other), the system has to choose between responding to requests (Availability) and returning the most recent data (Consistency). It can't do both, because if one server isn't reachable, the system can't guarantee that the data will be the same on all servers.
Conclusion: In the case of network partitions, you must choose Availability over Consistency or Consistency over Availability. Therefore, you can't have both consistency and availability without partition tolerance.

2. Consistency and Partition Tolerance without Availability (CP):

Scenario: Suppose you want to ensure that your system is always consistent (all data is synchronized across servers) and can still function if parts of the network fail (Partition Tolerance).
Problem: In the event of a network partition, some servers might be unreachable, but the system will still ensure that all nodes have consistent data by refusing to answer requests until the partition is resolved. This means Availability is sacrificed — the system might not respond to some requests because it is waiting for the partition to be resolved to keep consistency.
Conclusion: If the system guarantees Consistency and Partition Tolerance, it may not be able to respond to requests during a partition, thus sacrificing Availability.

3. Availability and Partition Tolerance without Consistency (AP):

Scenario: You want your system to respond to every request (availability) and keep working even if some parts of the network fail (partition tolerance).
Problem: If there is a network partition, different parts of the system may start serving different versions of the data because the system will continue to process requests even when some servers can't talk to each other. This can result in inconsistent data between nodes.
Conclusion: In this case, you give up consistency to ensure both Availability and Partition Tolerance.

Headless vs Front end application

System Design

Headless applications focus on separating the frontend (user interface) from the backend (server-side logic). This separation allows for greater flexibility and scalability, as the frontend and backend can be developed, maintained, and updated independently. In a headless application, the frontend communicates with the backend through APIs to fetch data and perform actions.

Frontend applications are traditional applications that have a tightly coupled frontend and backend. The frontend is responsible for rendering the user interface and handling user interactions, while the backend handles server-side logic and data management. Frontend applications typically have a fixed structure and are designed for a specific platform or device.

Key differences between headless and frontend applications:

Separation of concerns: Headless applications have a clear separation between the frontend and backend, while frontend applications have a more integrated approach.
Flexibility: Headless applications are more flexible as they can support multiple frontends and can be updated independently.
Scalability: Headless applications can be scaled more easily as the frontend and backend can be scaled independently.
Development: Headless applications often require more complex development processes due to the separation of concerns.
User experience: Frontend applications typically provide a more cohesive user experience as the frontend and backend are tightly integrated.

In summary

Frontend Application: This is a broad term that encompasses any application that interacts directly with the user. It could be a website, mobile app, or desktop application.
Headless Application: This is a specific architecture where the frontend (user interface) is decoupled from the backend (server-side logic) and communicates through APIs.

A React app that fetches data from an API falls into the headless category because it:

Separates frontend and backend: The React app (frontend) is distinct from the backend service providing the API.
Uses APIs: The React app communicates with the backend through APIs to retrieve and manipulate data.

In essence, while all React apps that use APIs are frontend applications, they also inherently follow a headless architecture due to the separation of concerns and API-based communication.

Headless application

System Design

A headless application is a software application that separates the front-end (user interface) from the back-end (server-side logic). This separation allows for greater flexibility and scalability, as the front-end and back-end can be developed, maintained, and updated independently.

Key characteristics of headless applications:

Decoupled front-end and back-end: The front-end and back-end components are separate entities that communicate through APIs.
API-driven: The back-end provides APIs that the front-end can use to fetch data and perform actions.
Multiple front-ends: A headless application can support multiple front-ends, such as web, mobile, and desktop applications.
Flexibility: This architecture allows for rapid changes to the front-end without affecting the back-end, and vice versa.
Scalability: The front-end and back-end can be scaled independently to meet changing demands.

Benefits of headless applications:

Faster development: The decoupled architecture allows for parallel development of the front-end and back-end.
Improved flexibility: The ability to update the front-end without affecting the back-end enables rapid changes and experimentation.
Enhanced scalability: The ability to scale the front-end and back-end independently ensures optimal performance and resource utilization.
Reusability: The back-end can be reused for multiple front-ends, reducing development effort.
Improved user experience: Headless applications can provide a more consistent and responsive user experience across different platforms.

Example:

A headless e-commerce application could have a back-end that manages product information, orders, and inventory. The front-end could be a web application, a mobile app, or even a voice-activated assistant. The front-end would communicate with the back-end through APIs to fetch product data, place orders, and manage the shopping cart.

API gateway

System Design

An API gateway acts as a single entry point for clients to access and interact with multiple backend services. It provides a unified interface, abstracting the complexity of the underlying systems and simplifying communication.

Key functions of an API gateway:

Unified interface: It presents a consistent API to clients, regardless of the backend services being accessed.
Authentication and authorization: It can handle authentication and authorization, ensuring that only authorized users can access specific resources.
Rate limiting: It can implement rate limiting to prevent abuse and protect backend services from overload.
Caching: It can cache frequently accessed data to improve performance and reduce load on backend services.
Transformation: It can transform data between different formats, ensuring compatibility between clients and backend services.
Security: It can provide security features like encryption, data validation, and protection against common attacks.

Benefits of using an API gateway:

Simplified development: It reduces the complexity of client-side development by providing a single endpoint.
Improved performance: It can improve performance by caching data and optimizing communication with backend services.
Enhanced security: It can strengthen security by implementing authentication, authorization, and rate limiting.
Scalability: It can handle increased traffic by distributing load across multiple backend services.
Flexibility: It can adapt to changes in backend services without affecting clients.

Example:
In a mobile application, an API gateway can act as a central point for clients to access different backend services, such as user authentication, product catalog, and order processing. The API gateway can handle authentication, rate limiting, and data transformation, providing a consistent and secure interface for the mobile app.

Load balancer

System Design

A load balancer can be a device or software that distributes incoming network traffic across multiple servers to improve performance, reliability, and scalability. It acts as a traffic manager, ensuring that workload is evenly distributed among available servers, preventing any single server from becoming overloaded.

Key functions of a load balancer:

Traffic distribution: It directs incoming traffic to the most appropriate server based on various factors, such as server load, health status, and application requirements.
Failover: If a server fails or becomes unresponsive, the load balancer can automatically redirect traffic to a working server, ensuring uninterrupted service.
Session persistence: It can maintain session affinity, ensuring that requests from a particular client are always routed to the same server, preserving the state of the session.
Performance optimization: Load balancers can improve performance by reducing latency, increasing throughput, and enhancing overall system responsiveness.
Security: Some load balancers can provide security features like SSL termination, DDoS protection, and access control.

Example:
In an e-commerce website, a load balancer can distribute incoming traffic from customers across multiple web servers, ensuring that the website remains responsive even during peak traffic periods. If one server fails, the load balancer can automatically redirect traffic to another server, preventing service interruptions.

FaaS (Function as a Service)

System Design

FaaS (Function as a Service) is a type of serverless computing where developers can write and deploy individual functions without having to manage the underlying infrastructure. These functions are typically small, single-purpose pieces of code that are triggered by events.

Key characteristics of FaaS:

Function-level granularity: You can deploy and manage functions independently, without having to package them into larger applications.
Event-driven: FaaS functions are triggered by events, such as HTTP requests, API calls, or messages from other services.
Automatic scaling: The platform automatically adjusts the number of instances based on demand, ensuring optimal performance and cost-efficiency.
Pay-per-use: You only pay for the resources consumed by your functions, rather than paying for a fixed server capacity.

Example:
Imagine you want to create a web application that processes images. Instead of setting up and managing a server to handle the image processing, you could use a FaaS platform to deploy a function that triggers when an image is uploaded. The function would then process the image and store the result in a cloud storage service.

Serverless

System Design

Serverless computing is a cloud computing model where developers can build, run, and manage applications without having to provision or manage servers. Instead of managing infrastructure, developers can focus on writing code and deploying it to a serverless platform.

Key characteristics of serverless computing:

Pay-per-use: You only pay for the resources consumed by your application, rather than paying for a fixed server capacity.
Automatic scaling: The platform automatically adjusts the number of instances based on demand, ensuring optimal performance and cost-efficiency.
Event-driven: Serverless functions are typically triggered by events, such as HTTP requests, API calls, or messages from other services.
Managed infrastructure: The platform handles all the underlying infrastructure, including servers, networking, and security.

Example:

Imagine you want to create a web application that processes images. Instead of setting up and managing servers to handle the image processing, you could use a serverless platform to deploy a function that triggers when an image is uploaded. The function would then process the image and store the result in a cloud storage service.