Big Data and Analytics

Big data refers to extremely large, fast-growing and varied collections of data that are too big or complex to be stored and processed by traditional database systems. It is usually described by the 5 Vs: volume, velocity, variety, veracity and value. Sources include social media, sensors, transactions and logs.

The characteristics of big data are the Vs.

• Volume: the huge size of data (terabytes to zettabytes).
• Velocity: the speed at which data is generated and processed.
• Variety: different forms such as structured, semi-structured and unstructured data.
• Veracity: the quality, accuracy and trustworthiness of data.
• Value: the useful insight that can be obtained from the data.

Some authors add variability and visualisation.

Big data integration is the process of combining data from many different sources and formats (databases, files, social media, sensors, logs, cloud applications) into a unified, consistent view for analysis. It uses extract-transform-load (ETL/ELT), data pipelines and tools such as Apache Kafka, Spark, Sqoop and Flume, and involves cleaning, mapping and standardising the data.

1. Define the business problem and objectives.
2. Data collection / acquisition.
3. Data storage.
4. Data cleaning and preprocessing (integration and transformation).
5. Data exploration and analysis (processing with analytic or machine-learning techniques).
6. Visualisation and interpretation of results.
7. Decision making, deployment and monitoring.

Big data analytics is the process of examining large and varied data sets to uncover hidden patterns, correlations, trends and customer preferences, so that organisations can make better decisions. It uses techniques such as data mining, statistics, machine learning and predictive modelling, with tools such as Hadoop, Spark and NoSQL databases. Its types are descriptive, diagnostic, predictive and prescriptive.

• HDFS: Hadoop Distributed File System for storage.
• YARN: resource management and job scheduling.
• MapReduce: distributed batch processing.
• Hive: SQL-like data warehouse queries.
• Pig: high-level data flow scripting.
• HBase: NoSQL column database.
• Sqoop: transfers data between Hadoop and relational databases.
• Flume: collects streaming log data.
• Spark: fast in-memory processing.
• Zookeeper: coordination service.
• Oozie: workflow scheduler.
• Mahout: machine learning. Others include Kafka and Ambari.

A NoSQL ("not only SQL") database is a non-relational database designed to store and retrieve large volumes of structured, semi-structured and unstructured data with flexible schema, horizontal scalability and high performance. The main types are document stores (MongoDB), key-value stores (Redis), column-family stores (Cassandra, HBase) and graph databases (Neo4j).

A data lake is a centralised repository that stores large amounts of raw data in its native format (structured, semi-structured and unstructured) at any scale, until it is needed for analysis. The schema is applied when the data is read (schema-on-read). It is commonly built on HDFS or cloud object storage such as Amazon S3.

A lakehouse is a modern data architecture that combines the low-cost, flexible storage of a data lake with the data management, ACID transactions, schema enforcement and fast SQL performance of a data warehouse. It lets business intelligence, machine learning and streaming workloads run on a single platform, for example using Delta Lake, Apache Iceberg or Databricks.

• Personalised product recommendations (Amazon, Netflix).
• Fraud detection in banking and insurance.
• Predictive healthcare, disease outbreak tracking and personalised medicine.
• Smart cities and traffic management.
• Precision agriculture and weather forecasting.
• Predictive maintenance in manufacturing.
• Sentiment analysis from social media.
• Targeted advertising and customer churn prediction.
• Self-driving vehicles and supply chain optimisation.
• Disaster response and public safety.

Comparison of big data and data warehouse

Benefits of big data

• Better and faster decision making based on evidence.
• Deeper understanding of customers and personalised service.
• Cost reduction and operational efficiency.
• Early fraud and risk detection.
• Product and service innovation and new revenue.
• Predictive analysis of trends and demand.
• Competitive advantage and real-time response.

Data quality in big data means that the data is accurate, complete, consistent, timely, valid and unique. Because data is large, fast and from many sources, quality must be managed systematically.

• Data governance: define ownership, policies, standards and responsibilities.
• Data profiling: examine data to find missing values, outliers, duplicates and format errors.
• Data cleansing: remove duplicates, correct errors, fill or treat missing values and standardise formats.
• Validation at entry: use rules, constraints and checks when data is captured or ingested.
• Standardisation and integration: common formats, units and definitions across sources; master data management.
• Metadata management and data lineage: record the origin and changes of data.
• Automated monitoring: quality metrics, alerts and audits in pipelines.
• Security and access control to prevent unauthorised change, and regular feedback and correction from users.

Big data analytics is applied in many fields.

• Retail and e-commerce: customer segmentation, recommendation engines, demand forecasting, dynamic pricing.
• Banking and finance: fraud detection, credit scoring, risk management, algorithmic trading.
• Healthcare: disease prediction, patient record analysis, drug discovery, personalised treatment.
• Telecommunications: churn prediction, network optimisation, location-based services.
• Manufacturing: predictive maintenance, quality control, supply chain optimisation.
• Government and smart cities: traffic management, crime prevention, tax compliance, disaster management, census analysis.
• Education: learner performance analysis and personalised learning.
• Agriculture: crop yield prediction and weather analysis.
• Media and marketing: sentiment analysis, targeted advertising, content recommendation.
• Transport and logistics: route optimisation and fleet tracking.

The data storage component of Hadoop is the Hadoop Distributed File System (HDFS). It stores very large files across many commodity machines in a cluster, providing high throughput and fault tolerance.

Architecture

• NameNode (master): keeps the metadata, i.e. the file system namespace, file-to-block mapping and the location of blocks. It controls access and block replication.
• DataNodes (slaves): store the actual data blocks, serve read and write requests and send heartbeats and block reports to the NameNode.
• Secondary NameNode / Standby NameNode: periodically merges the edit log with the file system image (checkpointing), or takes over in high-availability setups.

Working

• Files are split into large blocks (128 MB by default) and distributed across DataNodes.
• Each block is replicated (default replication factor 3) on different nodes and racks, so a failure of a node does not lose data.
• The write is once and read many times model; data is written sequentially, and the client contacts the NameNode for metadata and then reads or writes directly with DataNodes.

Features: scalability by adding nodes, fault tolerance, data locality (computing moves to data), low cost and suitability for batch processing of huge files.

NoSQL databases have these advantages.

1. Flexible schema: data can be stored without a fixed structure, so changes are easy and structured, semi-structured and unstructured data are all supported.
2. Horizontal scalability: capacity grows by adding commodity servers (sharding) rather than a costly bigger machine.
3. High performance: fast reads and writes for large volumes and simple access patterns, often with in-memory options.
4. High availability and fault tolerance: data is replicated and distributed across nodes, with no single point of failure.
5. Low cost: open-source options and commodity hardware reduce cost.
6. Suited to big data and real-time applications: such as social media, IoT, gaming and recommendation systems.
7. Variety of data models: document, key-value, column and graph models fit different problems.
8. Rapid development: the data model matches application objects (for example JSON), which reduces mapping effort and supports agile development.

A trade-off is that many NoSQL systems provide eventual consistency and limited joins compared with relational databases.

A data lake architecture is organised in layers, from raw ingestion to consumption.

1. Ingestion layer: brings data from many sources (databases, logs, IoT, social media, files) in batch or real time, using tools such as Kafka, Flume, Sqoop and NiFi.
2. Storage layer: holds the data in raw form at low cost on HDFS or cloud object storage (Amazon S3, Azure Data Lake Storage). It is often divided into zones: raw (landing), cleansed/curated, and consumption/sandbox zones.
3. Processing layer: transforms, cleans and analyses data in batch or stream using Spark, Hive, MapReduce or Flink.
4. Metadata and catalog layer: data catalogue, schema, data lineage and search, so users can discover and understand data.
5. Governance and security layer: access control, encryption, authentication, auditing and data quality management.
6. Consumption layer: BI tools, SQL query engines, machine learning, dashboards and APIs used by analysts and data scientists.

Data is stored as-is and the schema is applied on read.

Big data analytics is of four main types.

1. Descriptive analytics: answers "what happened?" by summarising historical data using reports, dashboards and aggregation, for example monthly sales.
2. Diagnostic analytics: answers "why did it happen?" by finding causes through drill-down, data mining and correlation, for example why sales fell in a region.
3. Predictive analytics: answers "what will happen?" using statistical models and machine learning to forecast, for example customer churn or demand.
4. Prescriptive analytics: answers "what should we do?" by recommending actions using optimisation and simulation, for example the best delivery route or price.

Big data itself is also classified by form as structured, semi-structured and unstructured data.

Advantages of data lakes

• Store all types of data (structured, semi-structured, unstructured) in raw form.
• Low-cost, highly scalable storage.
• Schema-on-read gives flexibility for future, unknown uses.
• Supports advanced analytics, machine learning and real-time processing.
• Breaks down data silos by keeping data in a single repository.
• Fast ingestion since no upfront transformation is needed.

Disadvantages of data lakes

• Without governance and metadata they become a "data swamp" that is hard to use.
• Data quality problems since data is stored uncleansed.
• Security and privacy risks from storing sensitive raw data in one place.
• Needs skilled data engineers and scientists.
• Query performance can be slower than a data warehouse, and ACID transactions are limited.
• Difficult data discovery and lineage tracking without proper cataloguing.

A data lake is a central repository that stores huge volumes of raw data in its native format (structured, semi-structured and unstructured) at low cost, and applies structure only when the data is read (schema-on-read).

Why it is important in big data systems

• Stores all data types: logs, clickstreams, images, video, sensor data and relational tables can sit together.
• Low-cost, scalable storage: built on commodity hardware or object storage such as HDFS and Amazon S3.
• Schema-on-read flexibility: data can be ingested now and interpreted later for different needs.
• Supports advanced analytics: machine learning, real-time and exploratory analysis can run directly on raw data.
• Breaks data silos: one place for data from many sources, available to many teams.

Data lake vs data warehouse

Social media platforms such as Facebook, YouTube, X and TikTok generate enormous volumes of posts, likes, shares, images and videos every second. Big data analytics turns this data into useful insight.

Importance

• Sentiment analysis: finds what people feel about a product, brand, event or political issue by analysing posts and comments.
• Targeted advertising: user interests, location and behaviour allow ads to be shown to the right audience, which raises conversion and revenue.
• Personalised recommendations: suggests friends, pages, videos and content (for example the YouTube and Facebook feeds).
• Trend detection: identifies trending hashtags and topics in real time, useful for marketing and news.
• Customer service and feedback: companies monitor complaints and respond quickly.
• Influencer identification: locates users with high reach and engagement for campaigns.
• Fake news and spam detection: patterns of abnormal activity help detect bots, hate speech and misinformation.
• Crisis and public-safety monitoring: posts during disasters help authorities understand needs and respond faster.
• Social network analysis: reveals communities and how information spreads.

Tools used include Hadoop, Spark, Kafka and NoSQL databases, which can handle the volume, velocity and variety of social data.

NoSQL ("not only SQL") databases store data in non-relational models. They scale horizontally, handle large volumes of varied data and have flexible schemas. There are four main types.

1. Key-value stores

Data is stored as a pair of a unique key and a value. Lookups by key are very fast, which suits caching, sessions and shopping carts. The value is opaque to the database.

Example: Redis, Amazon DynamoDB, Riak.

2. Document stores

Data is stored as self-describing documents (JSON, BSON or XML). Each document can have its own structure, and the database can query fields inside it. Suited to content management, catalogues and user profiles.

Example: MongoDB, CouchDB.

{ "_id": 1, "name": "Sita", "courses": ["BIM", "BBA"] }

3. Column-family (wide-column) stores

Data is stored in column families rather than rows, so reading a few columns across many rows is efficient and the store scales across clusters. Good for time series, logging and analytics on huge datasets.

Example: Apache Cassandra, HBase, Google Bigtable.

4. Graph databases

Data is stored as nodes (entities) and edges (relationships) with properties. They make relationship queries and traversals fast. Used for social networks, recommendation engines and fraud detection.

Example: Neo4j, Amazon Neptune.

Big data in health care

Health care produces large and varied data: electronic health records (EHR), medical images, lab results, wearable-device streams, insurance claims and genomic data. Analysing it helps in the following ways.

• Better diagnosis and treatment: machine learning on images (X-ray, MRI) and patient history helps detect diseases such as cancer and diabetic retinopathy early.
• Predictive analytics: predicts patients at risk of readmission, sepsis or chronic disease so that preventive care can be given.
• Personalised medicine: genomic and lifestyle data help choose the therapy that suits an individual.
• Disease outbreak tracking: analysing hospital, search and social data detects epidemics (for example COVID-19) early.
• Remote monitoring: wearables and IoT sensors stream vital signs, and alerts are raised in real time.
• Hospital operations: forecasting patient flow, bed and staff needs, and reducing costs and waiting time.
• Drug discovery: analysing clinical trial and molecular data shortens research time.

Big data in fraud detection

Banks, insurers and e-commerce firms process millions of transactions, which makes manual checking impossible.

• Real-time transaction monitoring: streaming tools (Kafka, Spark Streaming) score every card or mobile-wallet transaction as it happens.
• Anomaly detection: models learn a customer's normal behaviour (amount, location, time, device) and flag deviations.
• Machine learning classification: supervised models trained on past fraud cases identify new suspicious patterns.
• Graph analysis: reveals fraud rings through links between accounts, devices and addresses.
• Insurance and health claim fraud: detects duplicate, inflated or unusual claims.
• Benefits: lower financial loss, fewer false alarms, faster response and increased customer trust.

Challenges: privacy and security of patient and customer data, data quality, integration of different sources, and the need for skilled staff.