Cloud Computing

Cloud deployment models

A deployment model defines who owns the cloud infrastructure, who can access it and where it is located.

1. Public cloud. Resources are owned by a provider and shared by many tenants over the internet on a pay-as-you-go basis. Low cost, huge scalability, no maintenance; less control and weaker customisation. Examples: AWS, Microsoft Azure, Google Cloud.

2. Private cloud. Infrastructure is dedicated to a single organisation, hosted on-premises or by a third party. Gives high security, control and regulatory compliance, but needs high capital and operating cost. Examples: VMware vSphere, OpenStack deployments in banks or government.

3. Community cloud. Shared by several organisations with common concerns (security, compliance, mission), e.g. a group of universities or hospitals. Costs are shared, but capacity and governance are more limited than public cloud.

4. Hybrid cloud. A combination of two or more of the above, joined by technology that allows data and application portability. Sensitive workloads stay in the private cloud while peak load "bursts" to the public cloud. It is flexible but complex to manage and secure.

5. Multi-cloud (extension). Services from several public providers are used to avoid vendor lock-in.

Different cloud platforms

These platforms offer IaaS, PaaS and SaaS service models on top of the deployment models above.

VMware

VMware is a company (now part of Broadcom) and a family of virtualization products. Its products, such as ESXi, vSphere, vCenter, Workstation and Fusion, let one physical machine run many isolated virtual machines (VMs), each with its own operating system, CPU, memory, storage and network share.

Hypervisor

A hypervisor (Virtual Machine Monitor) is software or firmware layer that creates, runs and manages virtual machines on a physical host. It does the following:

• Abstracts and partitions the hardware (CPU, RAM, disk, NIC) among VMs.
• Schedules VM CPU time and allocates memory.
• Isolates VMs so that a crash or attack in one does not affect others.
• Provides management functions such as creation, snapshots, live migration and cloning.

Type 1 (bare-metal): runs directly on hardware, e.g. VMware ESXi, Xen, Hyper-V. Best performance; used in data centres. Type 2 (hosted): runs on a host OS, e.g. VMware Workstation, VirtualBox. Easier to set up; used on desktops.

Benefits of a hypervisor to an organisation

1. Server consolidation: many workloads on fewer servers, lowering hardware, power, cooling and space cost.
2. Better resource utilisation: idle capacity is shared instead of wasted.
3. Isolation and security: each VM is separate.
4. High availability and disaster recovery: snapshots, backups and live migration (vMotion) move VMs away from failing hardware without downtime.
5. Scalability and agility: new VMs are provisioned in minutes.
6. Testing and development: safe sandbox environments, and different OSes on the same hardware.
7. Easier management: centralised control of all VMs.

Advantages of cloud computing

• Cost saving: no upfront capital expense; pay only for what is used.
• Scalability and elasticity: resources grow or shrink with demand.
• Accessibility: services are reachable from anywhere with internet.
• Reliability and disaster recovery: replicated data centres and backups.
• Automatic updates and maintenance handled by the provider.
• Speed and agility: quick provisioning and deployment.
• Collaboration: shared access to data and applications.

Disadvantages

• Dependence on internet: no connectivity means no access.
• Security and privacy concerns: data is held by a third party.
• Vendor lock-in: moving between providers is difficult.
• Limited control over infrastructure and customisation.
• Downtime and outages of the provider affect all customers.
• Compliance/legal issues about data location.
• Hidden or growing costs for large data transfer and long-term use.

Detecting and eliminating intruders in a private cloud

Intrusion handling combines detection, response and removal:

1. Intrusion Detection Systems (IDS):
   • Network-based IDS monitors traffic between VMs and at the cloud boundary.
   • Host-based IDS monitors logs, file integrity and system calls on each VM or hypervisor.
   • Signature-based detection matches known attack patterns; anomaly-based detection flags behaviour that deviates from a normal baseline (unusual logins, traffic spikes, port scans).
2. Log collection and SIEM: logs from servers, firewalls and the hypervisor are correlated centrally to raise alerts.
3. Access monitoring: failed logins, privilege escalation and unusual access times are checked against IAM records.
4. Honeypots attract and study attackers.
5. Elimination (IPS and incident response):
   • Intrusion Prevention System blocks malicious packets automatically.
   • Compromised VMs are isolated/quarantined, their network access cut and their accounts or keys revoked.
   • Malware is removed, systems are restored from clean snapshots/backups, and vulnerabilities are patched.
   • Firewall rules and credentials are updated, and the incident is analysed to prevent recurrence.

Main configuration parameters in MapReduce

A MapReduce job is configured through the job configuration (Job/Configuration in Hadoop). The main parameters are:

1. Input location(s) and format: HDFS path of the input and the InputFormat (e.g. TextInputFormat) that splits data and reads records.
2. Output location and format: the output directory (must not already exist) and the OutputFormat.
3. Map class (Mapper): the function that converts input key-value pairs to intermediate key-value pairs.
4. Reduce class (Reducer): the function that merges all values of one key into the final result.
5. Combiner class (optional): a local mini-reducer that cuts the data shipped across the network.
6. Partitioner: decides which reducer receives which key (default is hash of key modulo number of reducers).
7. Data types: classes of input/output keys and values (e.g. Text, IntWritable).
8. Number of reducers (mapreduce.job.reduces) and the number of map tasks (determined by input splits).
9. JAR file / main driver class containing the job code.
10. Resource settings: memory and CPU per task, sort buffer size, block/split size, number of task retries and speculative execution.

Choosing between SOAP and REST

Choose SOAP when:

• enterprise-level security, reliable messaging and ACID transactions are needed (banking, payment, telecom);
• a formal contract (WSDL) between systems is required;
• the transport is not HTTP.

Choose REST when:

• building public, web and mobile APIs where simplicity, speed and scalability matter;
• bandwidth is limited and JSON is preferred;
• caching and stateless, loosely coupled services are desired.

In most modern cloud and mobile applications REST is preferred; SOAP remains in legacy and highly regulated systems.

Advantages of a data lake

A data lake is a central repository that stores large amounts of raw data in its native format (structured, semi-structured and unstructured) until it is needed.

1. Stores all data types: tables, logs, JSON, images, audio, video, sensor data.
2. Schema-on-read: structure is applied only when data is analysed, so ingestion is fast and flexible.
3. Low-cost scalable storage: built on cheap object storage or HDFS, scaling to petabytes.
4. Supports advanced analytics: machine learning, real-time analytics, data mining and big data processing on the same data.
5. Removes data silos: one place for data from many sources, giving a single view.
6. Preserves raw data: it can be reprocessed later for new questions.
7. Faster access for data scientists because they do not wait for ETL and modelling beforehand.
8. Integration with tools such as Spark, Hive, Presto and cloud services.

Backup strategy for systems, data and critical resources

A backup strategy ensures that data and systems can be restored after loss, corruption, attack or disaster.

1. Identify critical assets: classify systems, databases, configurations and applications by importance, and define RPO (acceptable data loss) and RTO (acceptable recovery time).
2. Backup types:
   • Full: a complete copy; slow but simple to restore.
   • Incremental: only changes since the last backup of any type; fast and small.
   • Differential: changes since the last full backup.
   • Snapshots/image backups for VMs and system state.
3. The 3-2-1 rule: keep 3 copies of data, on 2 different media, with 1 copy offsite (or in another cloud region).
4. Schedule and automation: regular, automated backups (e.g. weekly full, daily incremental), with continuous replication for critical databases.
5. Security: encrypt backups in transit and at rest, control access, and keep an immutable/offline copy against ransomware.
6. Storage location: on-site for fast restore, offsite or cloud for disaster protection.
7. Testing and verification: run periodic restore tests and integrity checks.
8. Retention and documentation: define how long copies are kept, and document the recovery procedure and responsibilities.

Thread programming vs task programming

Thread programming is a model in which an application is split into threads, the basic units of execution within a process, and the programmer creates, starts, synchronises and terminates them explicitly. Threads share memory, so locks, semaphores and monitors are needed to avoid race conditions and deadlocks. In the cloud it uses the threads of one (virtual) machine or a distributed thread abstraction such as Aneka threads.

Task programming expresses an application as a set of independent (or loosely dependent) tasks, each a unit of work such as a command or a program, that a runtime scheduler places on available resources. The programmer defines the tasks and their dependencies, not how they run; the middleware handles distribution, fault tolerance and scheduling. It suits bag-of-tasks applications and workflows.

Issues in fast data processing and streaming in the cloud

1. High velocity and volume: continuous, large-scale streams must be ingested without loss and with low latency.
2. Latency requirements: real-time decisions need millisecond to second response, hard over network hops and shared resources.
3. Scalability and elasticity: load fluctuates; scaling processing nodes without interrupting the stream is difficult.
4. Data quality: out-of-order, late, duplicate or missing events, and noisy data.
5. State management: maintaining windows and aggregates reliably across distributed nodes.
6. Fault tolerance and delivery guarantees: achieving at-least-once or exactly-once processing after failures.
7. Network bandwidth and cost: moving huge streams to the cloud is expensive and slow.
8. Security and privacy: protecting data in motion and meeting compliance rules.
9. Integration and heterogeneity: many sources, formats and tools (Kafka, Spark Streaming, Flink, Kinesis).
10. Resource contention and cost control: multi-tenancy performance variation and unpredictable bills.

Key components of an AWS architecture

• Compute: EC2 virtual servers, Lambda (serverless), ECS/EKS containers, Elastic Beanstalk.
• Storage: S3 object storage, EBS block storage, EFS file storage, Glacier archive.
• Database: RDS, DynamoDB, Redshift, ElastiCache.
• Networking and content delivery: VPC, subnets, route tables, Elastic Load Balancer, Route 53 DNS, CloudFront CDN.
• Security and identity: IAM, security groups, KMS, WAF, Shield.
• Management and monitoring: CloudWatch, CloudTrail, CloudFormation, Auto Scaling.
• Global infrastructure: Regions and Availability Zones for resilience and low latency.

Cloud governance

Cloud governance is the set of policies, roles, processes and controls used to manage cloud use so that it meets business, security and compliance goals. Key elements are:

• Policies and standards for usage, data classification and approved services.
• Roles and responsibilities (ownership, shared responsibility model).
• Security and compliance controls, audit and regulation requirements.
• Cost management through budgets, tagging and usage reports.
• Risk management and monitoring.
• Automation of guardrails with tools such as AWS Organizations, Config and Service Control Policies.

Parallel efficiency of MapReduce

Parallel efficiency measures how well added processors are used:

$S = \frac{T_1}{T_p}, \qquad E = \frac{S}{p} = \frac{T_1}{p\,T_p}$

where $T_1$ is the time on one node, $T_p$ the time on $p$ nodes, $S$ the speedup and $E$ the efficiency ($0<E\le 1$).

In MapReduce, map tasks work on independent input splits, so they parallelise almost perfectly and give high efficiency for large data. Efficiency drops because of:

• Shuffle and sort phase: moving intermediate data across the network.
• Reduce phase: reducers start after maps and may be limited by few keys (data skew gives stragglers).
• Framework overhead: job start-up, task scheduling and fault-tolerance (writing to disk).
• Stragglers: slow nodes delay the whole job (mitigated by speculative execution).
• Load imbalance between splits or keys.

Efficiency can be improved by data locality (moving computation to data), using combiners, a good partitioner, a suitable number of reducers and appropriate split size. Efficiency is best when the data is large relative to overhead, so it improves with problem size.

PaaS vs SaaS

PaaS sits between IaaS and SaaS: it removes infrastructure management so developers focus on code, whereas SaaS removes all management so users just use the software.

Role of the hypervisor in virtualization

A hypervisor (virtual machine monitor) sits between the hardware and the virtual machines. Its roles:

1. Abstraction: presents virtual CPU, memory, storage and network devices to each VM.
2. Resource allocation and scheduling: divides physical CPU time, RAM and I/O among VMs fairly.
3. Isolation: keeps VMs separate so faults and security breaches do not spread.
4. VM lifecycle management: create, start, pause, snapshot, clone and migrate VMs.
5. Hardware sharing and consolidation: runs several OSes on one machine.
6. Monitoring and control: tracks usage, enforces limits and supports live migration and high availability.

Type 1 hypervisors (ESXi, Xen, Hyper-V) run on bare metal; Type 2 (VirtualBox, VMware Workstation) run on a host OS.

Types of virtualization

• Hardware/server virtualization: one physical server is divided into multiple VMs (full virtualization, para-virtualization, hardware-assisted).
• Operating-system-level virtualization: multiple isolated user spaces (containers) share one kernel, e.g. Docker, LXC.
• Storage virtualization: several physical storage devices appear as one pool (SAN, NAS, RAID).
• Network virtualization: virtual networks (VLAN, VPN, SDN) abstracted from physical network hardware.
• Desktop virtualization: desktops run on a central server and are accessed remotely (VDI).
• Application virtualization: applications run in isolation from the underlying OS, e.g. Citrix, App-V.
• Data virtualization: data from different sources is accessed through one logical layer.

Serverless computing

Serverless computing is a cloud execution model in which the provider dynamically manages servers, scaling and capacity. Developers only write and upload code (usually small functions), and are billed per execution time and memory used, not for idle servers. Servers still exist, but they are hidden from the developer. Examples: AWS Lambda, Azure Functions, Google Cloud Functions.

Features: no server management, automatic scaling (down to zero), event-driven triggers, pay-per-use billing, and short-lived stateless functions (Function as a Service, FaaS).

Cloud deployment models

• Public cloud: owned by a provider, shared by many customers via the internet; cheap and scalable (AWS, Azure, GCP).
• Private cloud: exclusively for one organisation, on-premises or hosted; more control and security but higher cost.
• Community cloud: shared by organisations with common requirements (e.g. several banks or universities).
• Hybrid cloud: combination of public and private (or community) clouds, allowing data and application portability and cloud bursting.

Service

A service is a self-contained, reusable unit of software functionality that is offered over a network through a well-defined interface and can be discovered and invoked independent of platform and language. A web service is such a service exposed over web protocols, the basis of Service-Oriented Architecture (SOA).

SOAP-based web service model

SOAP (Simple Object Access Protocol) uses XML messages over HTTP/SMTP. The model has three roles:

1. Service provider: implements the service and publishes its description.
2. Service registry (UDDI): a directory where services are published and discovered.
3. Service requester (consumer): finds the service in the registry, fetches its WSDL and calls it.

Steps: the provider publishes a WSDL (Web Services Description Language) file describing operations, messages and endpoints to the UDDI registry; the requester finds it (find), binds to it (bind) and sends a SOAP message consisting of an Envelope containing an optional Header and a mandatory Body; the provider processes the request and returns a SOAP response.

Difference from RESTful architecture

Cloud computing security challenges and risks

Challenges

• Data security and privacy: data is stored off-premises under a provider's control; encryption and key management are essential.
• Loss of control and visibility over infrastructure.
• Multi-tenancy: shared hardware raises the chance of data leakage between tenants.
• Identity and access management: weak authentication and mismanaged privileges.
• Compliance and legal issues: data location and regulations (GDPR etc.).
• Interface and API insecurity: poorly secured APIs are attack points.
• Network security during transmission over the internet.

Risks

• Data breaches and data loss.
• Account or service hijacking.
• Insider threats, including malicious provider staff.
• Denial of Service (DoS/DDoS) attacks.
• Misconfiguration of cloud resources (open storage buckets).
• Vendor lock-in and provider failure or outage.
• Malware and insecure virtual machines or hypervisor vulnerabilities.
• Insecure data deletion and weak backup/recovery.

Graph database

A graph database stores data as a graph of nodes (entities), edges (relationships) and properties (attributes on both). Relationships are stored directly, so traversing connections is fast and does not need costly joins as in relational databases. Examples: Neo4j, Amazon Neptune, ArangoDB, JanusGraph. Queries use languages such as Cypher, Gremlin and SPARQL.

Applications

1. Social networks: friend suggestions, communities, influence analysis.
2. Recommendation engines: "customers who bought this also bought" in e-commerce and streaming.
3. Fraud detection: finding suspicious rings and unusual links among accounts and transactions.
4. Knowledge graphs and search: semantic search, as used by search engines and AI assistants.
5. Network and IT operations: impact analysis and root-cause finding in telecom and cloud networks.
6. Route and logistics optimisation: shortest paths in maps and supply chains.
7. Identity and access management: relationships between users, roles and resources.
8. Bioinformatics and drug discovery: protein and gene interaction networks.

Designing security architecture in the cloud

A cloud security architecture is designed in layers following the shared responsibility model (provider secures the cloud itself; customer secures what is in the cloud).

1. Requirements and risk assessment: identify assets, threats, compliance needs and data classification.
2. Identity and access management: strong authentication (MFA), role-based access, least privilege, single sign-on.
3. Network security: virtual private clouds, subnets, firewalls, security groups, network segmentation, VPN, IDS/IPS and DDoS protection.
4. Data security: encryption at rest and in transit, key management (KMS/HSM), data masking, backup and secure deletion.
5. Application security: secure coding, web application firewall, API protection, vulnerability scanning.
6. Infrastructure and host security: hardened VM images, patching, hypervisor and container security.
7. Monitoring, logging and incident response: central logging, SIEM, alerts, audit trails and a response plan.
8. Governance and compliance: policies, audits and standards such as ISO 27001.
9. Resilience: redundancy, availability zones, disaster recovery.

Principles: defence in depth, zero trust, least privilege and automation of security controls.

Elasticity

Elasticity is the ability of a cloud to automatically and rapidly add or remove resources (CPU, memory, instances, storage) to match the current workload, so that capacity follows demand closely. When traffic rises (e.g. an exam result day or a sale) resources scale out; when it falls they scale in. This avoids both under-provisioning (slow service) and over-provisioning (wasted money), and is typically implemented with auto-scaling groups and load balancers. Scaling may be vertical (bigger machine) or horizontal (more machines).

On-demand provisioning

On-demand (self-service) provisioning lets users obtain computing resources whenever they need them, instantly and without human interaction with the provider, usually through a web portal or API. Resources are released when no longer needed, and the user pays only for usage. It supports quick deployment, removes procurement delays and upfront capital cost, and works through virtualization and automation.

Together they give flexibility, cost efficiency and fast response to changing business needs.

Main characteristics of Platform as a Service (PaaS)

PaaS provides a ready environment for developing, testing, deploying and managing applications without handling the underlying infrastructure.

1. Development environment: built-in tools, programming languages, frameworks, libraries and IDE integration.
2. Managed infrastructure: the provider looks after servers, OS, storage, networking, patching and runtime.
3. Automatic scalability and load balancing of applications.
4. Middleware and services: databases, messaging, web servers, APIs and analytics as ready components.
5. Multi-tenancy: many users share the platform securely.
6. Support for the whole lifecycle: build, test, deploy, version and monitor in one place, with collaboration for teams.
7. Pay-as-you-go pricing and reduced cost and time to market.
8. Integration with other services and databases through web services.

Examples: Google App Engine, Heroku, AWS Elastic Beanstalk, Azure App Service.

MapReduce

MapReduce is a programming model and framework for processing large data sets in parallel across a cluster. A job has two user-written functions and a framework-managed shuffle:

• Map: reads input split records and emits intermediate pairs $(k_1, v_1) \rightarrow \text{list}(k_2, v_2)$.
• Shuffle and sort: the framework groups all values by key.
• Reduce: combines the values of each key: $(k_2, \text{list}(v_2)) \rightarrow \text{list}(k_3, v_3)$.

The framework handles splitting, scheduling near the data, and fault tolerance by rerunning failed tasks.

Example: word count

Input lines: "cloud is big", "big data is cloud".

• Map output: (cloud,1), (is,1), (big,1), (big,1), (data,1), (is,1), (cloud,1)
• After shuffle: big: [1,1]; cloud: [1,1]; data: [1]; is: [1,1]
• Reduce output: (big,2), (cloud,2), (data,1), (is,2)

def map(line):
    for w in line.split():
        emit(w, 1)

def reduce(word, counts):
    emit(word, sum(counts))

Fundamental principles of cloud security design

1. Defence in depth: multiple layers of controls (network, host, application, data) so one failure does not expose everything.
2. Least privilege: users and services get only the permissions they need.
3. Zero trust: never trust implicitly; verify every user, device and request.
4. Confidentiality, integrity and availability (CIA): encryption, hashing and access control, plus redundancy.
5. Separation of duties and isolation: segregate tenants, workloads and administrative roles.
6. Secure by default and fail-safe defaults: deny access unless explicitly allowed.
7. Strong identity and authentication: MFA and centralised identity management.
8. Shared responsibility: be clear which controls belong to provider and which to customer.
9. Continuous monitoring, logging and auditing with fast incident response.
10. Compliance, privacy and data protection by design.
11. Resilience and recoverability: backups, replication and disaster recovery.
12. Automation and minimal attack surface: infrastructure as code, patching, and disabling unneeded services.

Thread programming vs task programming

(a) Cloud stakeholders

Cloud stakeholders are the parties involved in providing, using and governing cloud services:

• Cloud service provider: owns and operates the infrastructure and delivers IaaS, PaaS or SaaS (AWS, Azure).
• Cloud consumer/customer: organisation or individual that uses the services.
• Cloud broker: intermediary who negotiates, aggregates and integrates services from several providers.
• Cloud auditor: independent party who assesses security, privacy and performance.
• Cloud carrier: network and telecom provider delivering connectivity.
• Developers/end users and regulators: build applications, use them, or enforce compliance.

(b) Fog computing

Fog computing extends cloud computing to the edge of the network. Processing, storage and networking are performed on fog nodes (routers, gateways, local servers) close to IoT devices rather than only in a distant data centre. Benefits: low latency, reduced bandwidth use, better real-time response, improved privacy and operation during poor connectivity. Applications: smart cities, connected vehicles, healthcare monitoring, industrial IoT. The architecture has three layers: device (things), fog and cloud.

Serverless computing

Serverless computing is a cloud model where the provider automatically manages servers, provisioning and scaling; developers deploy code and pay only for actual execution time. Servers exist but are invisible to the developer. Examples: AWS Lambda, Azure Functions, Google Cloud Functions.

How FaaS works

Function as a Service (FaaS) is the core of serverless:

1. The developer writes a small, stateless function and uploads it with its configuration.
2. The function is bound to an event trigger: an HTTP request through an API gateway, file upload, database change, message queue or timer.
3. When the event occurs, the platform starts a container to run the function (a cold start if none is ready; reused containers give warm starts).
4. The function executes, returns a result or writes output to storage or a database.
5. The platform scales instances automatically with the number of events, and shuts them down when idle.
6. The user is billed per invocation and execution time (milliseconds and memory).

Characteristics of cloud computing

• On-demand self-service.
• Broad network access.
• Resource pooling (multi-tenancy).
• Rapid elasticity and scalability.
• Measured service (pay-per-use).
• High availability and reliability.

Challenges of cloud computing

• Security and privacy of data.
• Compliance and legal issues.
• Vendor lock-in and interoperability.
• Downtime, performance and latency dependent on the internet.
• Limited control and visibility.
• Cost management, hidden costs.
• Data migration and integration with existing systems.
• Lack of skilled staff.

Virtualization

Virtualization is the creation of a virtual (software-based) version of a computing resource such as a server, operating system, storage device or network, allowing one physical resource to be shared by multiple isolated environments.

Significance in cloud computing

• It is the foundation of cloud: IaaS is built by running many VMs on shared hardware.
• Resource pooling and better utilisation of servers.
• Multi-tenancy and isolation between customers.
• Elasticity and rapid provisioning: VMs are created or removed in minutes.
• Cost reduction through consolidation of hardware, power and space.
• Flexibility and portability: VMs can be moved or live-migrated between hosts.
• Disaster recovery using snapshots and replication.
• Legacy support: different OSes on the same hardware.

How VMware and hypervisor work

VMware provides virtualization software. Its bare-metal hypervisor ESXi is installed directly on the physical server. The hypervisor:

1. Takes control of the hardware (CPU, RAM, disks, NICs).
2. Creates virtual machines, each with virtual hardware, and runs a guest OS inside. Privileged guest instructions are trapped and emulated or run using hardware-assisted virtualization (Intel VT-x, AMD-V).
3. Schedules CPU time and allocates memory (with techniques like ballooning and page sharing).
4. Provides virtual switches and datastores (VMFS) for network and storage.
5. Is managed centrally by vCenter Server, which offers vMotion (live migration), HA, DRS (load balancing) and snapshots.

A Type 2 hypervisor such as VMware Workstation instead runs as an application on a host OS.

Thread programming

Thread programming divides an application into threads, lightweight units of execution within a process that share memory and run concurrently, exploiting multiple cores or nodes. The programmer creates threads, assigns work, and synchronises them using locks, semaphores and barriers to avoid race conditions and deadlock. In cloud platforms (e.g. Aneka) a thread model allows the threads of an application to be distributed transparently over cloud resources, while the API resembles local thread programming (start, join, abort).

Example (Python):

from threading import Thread
results = [0]*4
def work(i, data):
    results[i] = sum(data)
data = [[1,2],[3,4],[5,6],[7,8]]
ts = [Thread(target=work, args=(i, data[i])) for i in range(4)]
for t in ts: t.start()
for t in ts: t.join()
print(sum(results))   # 36

MapReduce in the cloud programming model

MapReduce lets users process huge data sets on cloud clusters without handling threads or communication. The user writes only:

• map$(k_1,v_1)\rightarrow \text{list}(k_2,v_2)$
• reduce$(k_2,\text{list}(v_2))\rightarrow \text{list}(k_3,v_3)$

The runtime splits input across nodes, runs map tasks near the data, shuffles and sorts intermediate pairs by key, runs reducers, and re-executes failed tasks. Cloud services such as Hadoop on EC2 or Amazon EMR provide it on demand.

Example (word count): input "to be or not to be".

• Map: (to,1) (be,1) (or,1) (not,1) (to,1) (be,1)
• Shuffle: be:[1,1], not:[1], or:[1], to:[1,1]
• Reduce: (be,2) (not,1) (or,1) (to,2)

Thus MapReduce provides simple, scalable, fault-tolerant parallelism, whereas raw thread programming needs explicit synchronisation.

Grid computing vs fog computing

Cloud deployment models

• Public cloud: infrastructure owned by a provider and offered to the general public over the internet; pay-as-you-go, highly scalable, low cost but less control (AWS, Azure, Google Cloud).
• Private cloud: operated solely for one organisation, on-premises or hosted; high security, control and compliance but higher cost and maintenance.
• Community cloud: shared by several organisations with common concerns such as security or regulation (e.g. government bodies, universities); costs are shared.
• Hybrid cloud: combines public and private clouds, bound by technology that allows data and application portability; sensitive data stays private while peak loads use the public cloud (cloud bursting).
• Multi-cloud: uses services from multiple providers to avoid lock-in.

Choice depends on cost, security, control, compliance and scalability needs.

Server virtualization

Server virtualization divides one physical server into several isolated virtual servers.

1. Install a hypervisor: Type 1 (ESXi, Xen, KVM, Hyper-V) on bare metal.
2. The hypervisor abstracts CPU, RAM, disk and network into virtual hardware.
3. Create virtual machines, assigning vCPUs, memory, virtual disks and virtual NICs, and install guest operating systems.
4. Manage the VMs with a central tool (vCenter, OpenStack) for provisioning, snapshots, live migration and load balancing.

Techniques: full virtualization, para-virtualization and hardware-assisted virtualization (Intel VT-x/AMD-V). Containers offer OS-level alternative.

Storage virtualization

Storage virtualization pools physical storage from multiple devices so that it appears as a single logical store.

1. A virtualization layer (software, appliance or controller) sits between hosts and physical storage.
2. It maps logical volumes (LUNs) to physical blocks across disks and arrays (block-level) or maps logical files to physical locations (file-level).
3. It can be done host-based (volume managers), network-based (SAN virtualization appliance, in-band or out-of-band) or array-based (storage controller).
4. Features: thin provisioning, snapshots, replication, tiering and non-disruptive data migration.

Benefits: better utilisation, simplified management and flexible expansion.

Developing web services with SOAP

1. Define the service interface: write the WSDL describing operations, input/output messages, data types (XSD) and endpoint.
2. Implement the service in a language such as Java (JAX-WS) or .NET (WCF), generating server stubs from the WSDL.
3. Publish the service on a server and optionally register it in a UDDI registry.
4. Client obtains the WSDL, generates a proxy and sends a SOAP request:

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">
  <soap:Body>
    <getPrice><item>Book</item></getPrice>
  </soap:Body>
</soap:Envelope>

5. The server processes it and returns a SOAP response (or Fault) over HTTP.

Developing web services with REST

1. Identify resources and give each a URI, e.g. /books/12.
2. Map operations to HTTP methods: GET (read), POST (create), PUT (update), DELETE (remove).
3. Implement with frameworks such as JAX-RS, Spring Boot, Flask or Express; return representations in JSON or XML with proper status codes (200, 201, 404).
4. Keep the service stateless and cacheable.
5. Clients call the URL directly:

GET /books/12  ->  {"id":12,"title":"Cloud Computing","price":450}

SOAP is contract-driven and heavy; REST is simple and lightweight.

Cloud availability

Availability is the proportion of time a service is accessible, e.g. $A=\frac{\text{MTBF}}{\text{MTBF}+\text{MTTR}}$; "99.99%" allows about 52 minutes of downtime per year. It is achieved by:

• Redundancy: duplicate servers, storage and network paths.
• Multiple Availability Zones and regions so a data centre failure does not stop service.
• Load balancing and health checks that route traffic away from failed nodes.
• Auto scaling and auto healing replace failed instances.
• Data replication (synchronous or asynchronous) across zones.
• Live migration and clustering of VMs.
• Monitoring and SLAs.

Disaster recovery

Disaster recovery (DR) restores systems after major failures such as natural disasters, cyber attacks or human error. Steps:

1. Define RPO (tolerable data loss) and RTO (tolerable downtime) via business impact analysis.
2. Regular backups and snapshots, stored offsite or in another region.
3. Replication of data and VMs to a secondary site.
4. DR strategies by cost and speed: backup and restore, pilot light (minimal core running), warm standby (scaled-down copy), and multi-site active-active.
5. Failover to the DR site and failback after recovery, using automation (e.g. AWS Elastic Disaster Recovery, Azure Site Recovery).
6. Testing the DR plan regularly and documenting procedures.

Cloud security architecture

Cloud security architecture is the design of controls, policies and technologies that protect cloud data, applications and infrastructure. It is layered and follows the shared responsibility model.

Main layers and components

1. Physical and infrastructure security: secured data centres, hardware, redundant power (provider responsibility).
2. Network security: firewalls, virtual private cloud, segmentation, VPN, IDS/IPS, DDoS protection.
3. Identity and access management (IAM): authentication, MFA, role-based access, single sign-on.
4. Data security: encryption in transit and at rest, key management, data loss prevention, backup.
5. Virtualization/host security: hypervisor hardening, VM isolation, patching, container security.
6. Application security: secure development, WAF, API security, vulnerability testing.
7. Security management: logging, monitoring, SIEM, incident response, auditing.
8. Governance, risk and compliance: policies, standards (ISO 27001), privacy rules.

Principles: defence in depth, least privilege, zero trust and continuous monitoring, giving confidentiality, integrity and availability.

Identity management and access control in cloud security

Identity management (IdM) establishes and maintains digital identities of users, devices and services:

• Provisioning and deprovisioning: creating accounts when users join and removing them when they leave.
• Authentication: verifying identity by passwords, multi-factor authentication, biometrics, certificates or tokens.
• Single sign-on (SSO) and federation: one login across services using standards such as SAML, OAuth 2.0 and OpenID Connect.
• Directory services: central user stores such as Active Directory or cloud IAM (AWS IAM, Azure AD).

Access control decides what an authenticated identity may do (authorisation):

• Role-based (RBAC): permissions assigned to roles, users assigned to roles.
• Attribute-based (ABAC): decisions from user, resource and context attributes.
• Discretionary/Mandatory models according to owner or policy.
• Least privilege and separation of duties in policies.
• Policies and tokens attached to users, groups or services, with temporary credentials.

Supporting practices: logging and auditing of access, periodic access reviews, privileged access management and conditional access (based on location or device).

Row storage

In a row-oriented store, all fields of one record are stored together contiguously on disk. For a table with columns (ID, Name, Marks):

[1, Ram, 80] [2, Sita, 90] [3, Hari, 70]

• Efficient for reading or writing a whole record at once.
• Suited to OLTP workloads: inserts, updates, point lookups (MySQL, PostgreSQL, Oracle).
• Inefficient for analytics, since reading one column requires scanning all columns of every row.

Columnar storage

In a column-oriented store, the values of each column are stored together:

ID:    1, 2, 3
Name:  Ram, Sita, Hari
Marks: 80, 90, 70

• Queries like AVG(Marks) read only the needed column, reducing I/O.
• Similar values together give high compression (run-length, dictionary encoding) and fast vectorised processing.
• Suited to OLAP, data warehouses and big data analytics (Redshift, BigQuery, Parquet, Cassandra-like column families).
• Inserting or updating a single full row is slower, because several column files must be touched.

Graph database

A graph database stores data as nodes (entities), edges (relationships between nodes) and properties (key-value attributes). Relationships are first-class and stored directly, so connected data is traversed fast without joins. Examples: Neo4j, Amazon Neptune. Queried with Cypher, Gremlin or SPARQL.

Graph processing

Graph processing analyses large graphs $G=(V,E)$ to extract insight from structure. It is done in two ways:

1. Graph database queries (online): traversals and pattern matching, e.g. find friends-of-friends, shortest path.
2. Graph analytics (offline, large-scale): distributed frameworks process the whole graph with iterative algorithms:
   • Pregel-style "think like a vertex" model: computation proceeds in supersteps; in each, a vertex receives messages, updates its state, and sends messages to neighbours; the process halts when all vertices vote to halt (BSP model).
   • Implementations: Apache Giraph, Spark GraphX, GraphLab, Neo4j Graph Data Science.

Common algorithms: PageRank, shortest path, connected components, community detection, centrality and triangle counting. Graph is partitioned across cluster nodes (edge-cut or vertex-cut) to scale on the cloud.