Multimedia System Application

Multimedia is the integrated use of multiple media types, such as text, graphics, images, audio, video and animation, delivered and controlled by a computer, usually interactively.

Morphing is an animation technique that smoothly transforms one image or shape into another by interpolating between corresponding points, producing a gradual change of form and colour.

MIDI (Musical Instrument Digital Interface) is a standard protocol for communication between electronic musical instruments and computers. It transmits instructions (notes, pitch, duration, volume) rather than actual sound samples, so MIDI files are very small.

Digital audio is sound represented as a sequence of binary numbers obtained by sampling an analog sound wave at regular intervals and quantising each sample, e.g. CD audio at 44.1 kHz and 16 bits.

An authoring tool is software used to create multimedia applications by combining and sequencing text, graphics, audio, video and animation and adding interactivity, usually without detailed programming. Examples: Adobe Animate, Macromedia Director, Authorware.

Lossy compression reduces file size by permanently discarding some information that is less noticeable to human perception, so the original data cannot be restored exactly. Examples: JPEG, MP3, MPEG.

Speech analysis is the process of examining a speech signal to extract its features, such as pitch, intensity and spectrum, in order to identify the speaker, recognise the words or understand the speech content.

A medium is a means by which information is represented, stored, transmitted or presented, e.g. text, graphics, image, audio, video; it may be perception, representation, presentation, storage or transmission media.

JPEG (Joint Photographic Experts Group) is an international standard for compressing continuous-tone still images, mainly using lossy DCT-based compression, with a lossless mode also defined.

Dithering is a technique that simulates colours or grey levels not available in a limited palette by arranging dots of available colours in patterns, so that the eye blends them into intermediate shades.

A multimedia system has the following main components:

• Capture devices: keyboard, mouse, microphone, scanner, digital camera, video camera, used to input media.
• Storage devices: hard disk, CD/DVD, flash memory and servers that hold the large media files.
• Communication networks: LAN, WAN, Internet, used to transmit multimedia data with adequate bandwidth.
• Computer system: a fast processor, large memory, sound and graphics cards to process media.
• Display and output devices: monitor, speakers, projector, printer.
• Software: operating system, authoring tools, editing tools and compression codecs that create and present content.

Baseline JPEG compression is a lossy, sequential DCT-based process.

Source image (8x8 blocks) -> FDCT -> Quantizer -> Entropy encoder -> Compressed data
                                        ^               ^
                                  Quantization     Huffman tables
                                     table

1. Block preparation: the image is split into 8x8 pixel blocks (colour images are first converted to YCbCr, with chroma optionally subsampled), and sample values are level-shifted by subtracting 128.
2. Forward DCT (FDCT): each block is transformed into 64 DCT coefficients: one DC coefficient and 63 AC coefficients, concentrating energy in low frequencies.
3. Quantization: each coefficient is divided by the corresponding entry of a quantization table and rounded. This is the lossy step; high-frequency values mostly become zero.
4. Entropy coding: the DC coefficients are coded by differences from the previous block (DPCM); AC coefficients are read in a zigzag order, run-length coded, and then Huffman coded.

Decoding reverses the steps: entropy decoding, dequantization, inverse DCT.

Characteristics of a multimedia system:

• Computer controlled: a computer produces, processes, stores and presents the media.
• Integrated: several media types are combined into one system.
• Digital: all media are represented in digital form.
• Interactive: users can control the flow and content of the presentation.
• Independent media: each medium keeps its own type, but they are coordinated.
• Synchronised and real-time: time-dependent media such as audio and video must be synchronised and delivered with limited delay.
• Large storage and bandwidth demand: requires compression and fast processing.

Digital image representation needs two major processes:

1. Sampling (spatial digitisation): the continuous image is measured at a regular grid of points, giving a matrix of picture elements (pixels); more samples give higher resolution.
2. Quantization: the measured brightness or colour of each pixel is mapped to one of a finite set of levels, stored with a fixed number of bits (e.g. 8 bits give 256 levels).

Together they convert an analog image to an array of numbers, which is then stored, optionally compressed, and displayed. The storage size is $\text{width} \times \text{height} \times \text{bits per pixel}$.

Methods of controlling animation:

• Explicit control: the animator specifies every motion directly, such as translation, rotation and scaling, or gives the key frames, and the system interpolates the in-betweens.
• Procedural control: motion is described by procedures or rules; the state of one object affects others (e.g. a particle system following physical laws).
• Constraint-based control: motion is governed by constraints like fixed joints or connections between objects.
• Tracking live action: motion of a real actor or object is captured (e.g. rotoscoping, motion capture) and applied to the animated character.
• Kinematic and dynamic control: kinematics specifies positions and velocities (forward/inverse kinematics), dynamics uses forces and masses to compute the motion.
• Actor-based control: each object is an actor with behaviour.

The lossy sequential DCT-based mode (baseline JPEG) encodes an image in a single left-to-right, top-to-bottom scan.

Main steps:

1. The image is divided into 8x8 blocks and level-shifted.
2. A forward DCT converts each block into 64 coefficients (1 DC and 63 AC).
3. The coefficients are quantized with a quantization table; this discards information and gives the compression.
4. DC coefficients are DPCM coded; AC coefficients are zigzag scanned, run-length coded and Huffman coded.

It is simple, widely supported and gives high compression with good quality for photographs. The image is reconstructed fully only after the decoder reads the entire scan, and quality loss grows with coarser quantization.

Toolkits are sets of software tools and libraries used to build multimedia applications.

• Graphics and image tools: Adobe Photoshop, CorelDRAW, GIMP for creating and editing images.
• Audio tools: Audacity, Sound Forge for recording, editing and mixing sound.
• Video editing tools: Adobe Premiere, Final Cut Pro for editing and combining clips.
• Animation tools: Adobe Animate, Maya, 3ds Max for 2D/3D animation.
• Authoring tools: Director, Authorware, Flash that combine media and add interaction.
• Programming toolkits and APIs: DirectX, OpenGL, Java Media Framework, QuickTime SDK, for developers to handle media playback, graphics and capture.

Multimedia affects society in many positive and negative ways.

Positive impacts

• Education: e-learning, simulations and interactive lessons make learning attractive and available at a distance.
• Entertainment: games, films, music and streaming.
• Business and communication: video conferencing, advertising, online shopping and presentations.
• Healthcare: medical imaging, telemedicine and training simulators.
• Information access: news, digital libraries and public awareness campaigns reach many people quickly.
• Employment: new jobs in design, animation and media production.

Negative impacts

• Addiction to games and social media, and reduced physical activity.
• Spread of misinformation, fake or manipulated media (deepfakes).
• Privacy, copyright and piracy issues.
• Exposure to inappropriate content and the digital divide between those with and without access.

Companies can use multimedia applications to gain competitive advantage in the following ways:

• Marketing and advertising: attractive video, animation and interactive ads attract and retain customers and build brand image.
• Customer service: interactive kiosks, online product demos and video support improve satisfaction.
• Training: multimedia-based e-learning cuts training cost and time for employees.
• Communication and collaboration: video conferencing reduces travel and speeds up decisions among distributed teams.
• Product design and presentation: 3D models and virtual prototypes shorten development time, and presentations help win clients.
• E-commerce: rich images, video and virtual try-ons increase sales.
• Better decision making: visualised data and multimedia reports make information easier to understand.

By lowering costs, improving quality and reaching customers faster, the company differentiates itself from competitors.

Lossless compression is a data compression method in which the original data can be reconstructed exactly and without any loss from the compressed data. It works by removing statistical redundancy, and is required where every bit matters, such as text, program files, medical images and data records.

Examples of techniques: Run-Length Encoding (RLE), Huffman coding, Lempel-Ziv-Welch (LZW), arithmetic coding. File formats: ZIP, PNG, GIF, TIFF (LZW).

Example (RLE): the string AAAABBBCC is stored as 4A3B2C. Decompression gives back exactly AAAABBBCC, with 9 symbols reduced to 6.

Compression ratio is lower than with lossy methods, because nothing is discarded.

Benefits

• Makes information more engaging, easier to understand and remember, because several senses are used.
• Supports interactivity and personalised learning.
• Useful in education, training, entertainment, business and healthcare.
• Enables communication at a distance (video conferencing, streaming).
• Presents complex ideas through animation and simulation.

Challenges

• Large storage and bandwidth requirements: media files are big and need compression and fast networks.
• Synchronisation and real-time delivery of audio and video.
• High cost of hardware, software and skilled development.
• Compatibility of formats and platforms.
• Copyright, security and piracy issues.
• Poor design can distract or overload the user.

Colour dithering

Many displays can show only a limited number of colours (e.g. a 256-colour palette). Colour dithering simulates colours missing from the palette by mixing nearby pixels of available colours, so that from a normal viewing distance the eye blends them into the desired colour. It extends grey-level halftoning to colour images.

Common methods:

• Ordered dithering: each pixel is compared with a threshold from a repeating dither matrix (e.g. Bayer matrix), applied for each colour channel. It is fast and gives a regular pattern.
• Error diffusion (Floyd-Steinberg): the quantization error of a pixel is spread to neighbouring unprocessed pixels (weights 7/16, 3/16, 5/16, 1/16), giving a more natural result at greater cost.
• Random dithering: noise added before thresholding; it avoids patterns but looks grainy.

Steps: choose the palette, for each pixel find the nearest palette colour (after adding the dither pattern or diffused error), then output it. Benefits: better appearance with few colours and smaller files. Drawbacks: reduced sharpness and visible patterns or noise.

RGB vs CMY colour models

Conversion between them is $(C, M, Y) = (1,1,1) - (R, G, B)$ with values normalised to $[0, 1]$.

Video conferencing

Video conferencing lets people at different places see and hear each other in real time using cameras, microphones, codecs, displays and a network.

Applications:

• Business meetings and collaboration without travel, saving time and cost.
• Distance education: live lectures and virtual classrooms.
• Telemedicine: remote consultation and diagnosis.
• Interviews, customer support, legal hearings and government meetings.
• Group discussions with screen and file sharing.

Requirements: real-time compression (H.261/H.263/H.264), low delay, synchronised audio and video, and enough bandwidth. Advantages: lower travel cost and faster decisions. Problems: bandwidth dependence, delay and quality issues, and security.

Video on demand (VoD)

VoD lets users select and watch video content whenever they like, with VCR-like control (play, pause, rewind, fast forward), unlike broadcast TV at fixed times. A video server stores compressed (MPEG) videos and streams them over the network to the user's set-top box or computer.

Applications:

• Entertainment: movies, TV shows, music videos (e.g. Netflix, YouTube).
• Education: recorded lectures and training videos on demand.
• News on demand and corporate training.
• Interactive TV, video libraries, and e-commerce product demos.

Requirements: large server storage, high network bandwidth, QoS guarantees, support for many simultaneous users and billing. Advantage: viewing is flexible and personalised. Problems: heavy bandwidth load, and copyright protection.