Multimedia Computing

Global structure of multimedia

A multimedia system handles several media (text, graphics, images, audio, video, animation) that are digitally captured, stored, processed, transmitted and presented in an integrated way. Its global structure is a layered organisation of components from the user down to the hardware.

+------------------------------------------+
|  Applications (education, games, VOD)    |
+------------------------------------------+
|  User interface / presentation layer     |
+------------------------------------------+
|  Authoring tools and programming layer   |
|  (abstraction levels, APIs, toolkits)    |
+------------------------------------------+
|  Media processing and compression        |
|  (JPEG, MPEG, audio coders, graphics)    |
+------------------------------------------+
|  Operating system, device drivers,       |
|  synchronisation and scheduling          |
+------------------------------------------+
|  Storage (HDD, CD/DVD, flash) and        |
|  network / communication subsystem       |
+------------------------------------------+
|  Input/output hardware: camera, mic,     |
|  scanner, display, speakers              |
+------------------------------------------+

Components

• Input and capture devices: keyboard, mouse, scanner, digital camera, microphone, video capture card convert real-world media into digital form (sampling and quantisation).
• Processing unit: a fast CPU with GPU and DSP support for compression, rendering and effects.
• Storage: large capacity and high throughput storage, since media is big (uncompressed video is of the order of hundreds of Mbit/s).
• Communication subsystem: networks that provide the bandwidth, low delay and low jitter needed for streaming and conferencing.
• Output devices: monitor, speakers, projector, printer.
• Software: the operating system with real-time scheduling, media drivers, compression codecs, authoring tools (for example Flash, Director) and application software.
• User interface: presents and lets the user navigate and interact with the integrated media.

Flow. Media is captured, digitised, compressed, stored or transmitted, decompressed, synchronised and then presented to the user. Synchronisation of the time-dependent (continuous) media such as audio and video with the discrete media (text, image) is the key feature that separates a multimedia system from a collection of separate media tools.

JPEG compression process

JPEG (Joint Photographic Experts Group) is a lossy, DCT-based standard for continuous-tone still images. The baseline encoder has these steps.

1. Colour space conversion and subsampling. RGB is converted to $YC_bC_r$ (luminance Y and chrominance $C_b, C_r$). The eye is less sensitive to colour detail, so $C_b$ and $C_r$ are subsampled (for example 4:2:0), which already halves the data.

2. Block splitting. Each component is divided into $8\times 8$ blocks, and 128 is subtracted from every sample (level shift) so that values lie in $-128\ldots127$.

3. Forward DCT. Each block is transformed:

$F(u,v)=\tfrac14 C(u)C(v)\sum_{x=0}^{7}\sum_{y=0}^{7} f(x,y)\cos\frac{(2x+1)u\pi}{16}\cos\frac{(2y+1)v\pi}{16}$

with $C(k)=1/\sqrt2$ for $k=0$ and $1$ otherwise. $F(0,0)$ is the DC coefficient (block average) and the other 63 are AC coefficients. Energy gets concentrated in the low-frequency, top-left coefficients. This step is nearly lossless.

4. Quantisation (the lossy step). Each coefficient is divided by the matching entry of a quantisation table $Q(u,v)$ and rounded:

$F_q(u,v)=\operatorname{round}\!\left(\frac{F(u,v)}{Q(u,v)}\right)$

Q has small values near the top-left and large values for high frequencies, so most high-frequency coefficients become 0. Example: $F(0,0)=-415$, $Q=16$ gives $-415/16=-25.9\to-26$; $F(0,1)=-30$, $Q=11$ gives $-2.7\to-3$; $F(4,3)=-2$, $Q=51$ gives $-0.04\to0$.

5. Zig-zag scan. The $8\times8$ quantised block is read in zig-zag order from the top-left, giving a 1-D sequence with the low-frequency values first and a long run of zeros at the end, for example: $-26, -3, 0, -3, -2, -6, 2, -4, 1, \ldots, 0,0,0,0$.

6. Entropy coding.

• DC coefficients are coded by DPCM: only the difference from the previous block's DC is coded, since neighbouring blocks have similar averages.
• AC coefficients are run-length coded as (run of zeros, value) pairs, ending with an EOB (end of block) code. Example: $(0,-3),(1,-3),(0,-2),\ldots$, EOB.
• The pairs and DC differences are then Huffman (or arithmetic) coded.

7. Bit stream. The coded data with headers, tables and markers forms the JPEG file.

Decoding reverses the steps: entropy decode, de-zigzag, multiply by $Q$ (dequantise), inverse DCT, add 128, upsample chroma and convert to RGB. The quantisation error cannot be recovered, hence JPEG is lossy. A larger quantisation table gives a higher compression ratio and lower quality.

Huffman coding

Huffman coding is a lossless, variable-length, prefix-free entropy coding method. Symbols that occur often are given short code words and rare symbols get long ones, so the average code length approaches the entropy of the source. No code word is a prefix of another, so decoding is unique and instantaneous.

Procedure

1. Count the frequency (or probability) of each symbol.
2. Make each symbol a leaf node and put all nodes in a list ordered by frequency.
3. Repeatedly remove the two nodes of lowest frequency and join them under a new parent whose frequency is their sum; put the parent back in the list.
4. Stop when one node (the root) is left.
5. Label each left branch 0 and each right branch 1. The code of a symbol is the sequence of labels from the root to its leaf.

Example. Symbols and frequencies in a 100-symbol message: A = 5, B = 9, C = 12, D = 13, E = 16, F = 45.

Merging:

• A(5) + B(9) = 14
• C(12) + D(13) = 25
• 14 + E(16) = 30
• 25 + 30 = 55
• F(45) + 55 = 100 (root)

Assigning 0 to F's branch and 1 to the 55 branch, then 0 to the 25 branch and 1 to the 30 branch, and so on:

Total bits $=45(1)+12(3)+13(3)+16(3)+5(4)+9(4)=45+36+39+48+20+36=224$ bits.

Average length $=224/100=2.24$ bits/symbol, compared with 3 bits/symbol for fixed-length coding of 6 symbols (300 bits). The saving is $(300-224)/300\approx25\%$.

Decoding example: the bit string 0 111 100 reads as F, E, C.

Huffman coding is used in JPEG, MPEG, and in ZIP (Deflate). Its limitation is that the code table must be sent with the data (or a fixed table agreed on), and it uses a whole number of bits per symbol.

Properties of multimedia computing

A multimedia computing system has the following properties.

• Combination of media: it integrates at least one continuous (time-dependent) medium, such as audio, video or animation, with discrete media such as text, graphics and images.
• Computer-controlled: all media are generated, processed, stored and presented under computer control.
• Digital representation: every medium is held in digital form, so it can be copied without loss, edited, compressed and sent over networks.
• Integration and independence of media: the media are combined in one system, but each can still be handled separately, and the same data can be used on different platforms and devices.
• Interactivity: the user decides what to see, when and in which order (hypermedia navigation, menus, simulations), instead of being a passive viewer.
• Synchronisation: the temporal and spatial relations between media, for example lip movement with speech, must be maintained within tight limits.
• High data volume and real-time demand: media need large storage and bandwidth, so compression, fast processors and real-time scheduling with a guaranteed quality of service are essential.
• Communication and sharing: networked multimedia supports streaming, conferencing and collaboration.

Bitmap and vector graphics

Computer-based animation

Animation is the illusion of movement created by displaying a sequence of still images (frames) rapidly enough, usually 12 to 30 frames per second, so that persistence of vision merges them into continuous motion. Computer-based animation uses the computer to create, store and play these frames.

Basic steps

1. Storyboard and input process: the idea is planned and key drawings are made or digitised.
2. Composition stage: foreground and background are combined.
3. In-betweening: the computer generates the intermediate frames between key frames, by interpolation.
4. Colouring and rendering: colour, texture, lighting and shading are applied and the final frames are produced.
5. Output and playback: the frames are stored as a video or animation file and played.

Types

• Cel (frame-by-frame) animation: each frame is drawn separately.
• Key-frame animation: only key frames are given and the software interpolates in between.
• Path animation: an object moves along a defined path (motion path).
• Morphing: one image transforms smoothly into another.
• 2-D and 3-D animation: 3-D uses models, skeletons, texturing and rendering.
• Procedural and physics-based animation: motion is computed from rules or physical laws.
• Motion capture: real actor movement is recorded and applied to a model.

Applications: films and cartoons, games, education and simulation, advertising, and scientific visualisation. Common tools include Flash, Maya, 3ds Max and Blender.

Speech generation

Speech generation (speech synthesis) produces spoken output from a computer. The aim is both natural sound and understandable content.

Basic notions. Speech is made of phonemes (basic sound units). Each sound has a fundamental frequency (pitch), formants (resonance frequencies of the vocal tract) and duration. Prosody means the rhythm, stress and intonation.

Methods

1. Reproduce stored speech (time-dependent sound concatenation). Words or sentences are recorded and stored digitally (for example PCM) and played back. Quality is high but the vocabulary is limited by storage.
2. Sound concatenation (phoneme, diphone, syllable). Basic units are stored and joined to make any word. The problem is the transition between units, so the diphone (the part from the middle of one phoneme to the middle of the next) is used to keep the coarticulation effects.
3. Frequency-dependent sound concatenation (formant synthesis). The vocal tract is modelled as a set of filters excited by a pulse train (voiced sounds) or noise (unvoiced sounds). Formant frequencies are changed over time to produce each sound. It needs little memory but sounds artificial.

Text-to-speech steps

1. Text analysis: normalise numbers and abbreviations, and convert letters to phonemes (grapheme-to-phoneme rules and dictionary).
2. Prosody generation: decide pitch, duration and stress.
3. Waveform synthesis: produce the audio signal by one of the above methods.
4. Output: digital-to-analogue conversion and the speaker.

Applications include screen readers for the blind, voice response systems, navigation and talking assistants.

Multimedia interface components

A multimedia user interface lets the user control and perceive several media. Its main components are:

• Input devices: keyboard, mouse, touch screen, pen, joystick, microphone (voice input), scanner, camera.
• Output devices: high-resolution display, speakers or headphones, projector, printer, and for virtual reality head-mounted displays.
• Visual interface elements: windows, menus, icons, buttons, dialogue boxes, scroll bars and the pointer (the WIMP style) used to navigate and select.
• Navigation structures: hyperlinks, hotspots, linear, hierarchical and web structures that guide the user through the content.
• Media players and controls: play, pause, rewind, volume and seek controls for audio, video and animation.
• Audio interface: speech recognition, speech synthesis and sound cues (earcons and auditory icons).
• Interaction techniques: direct manipulation, drag and drop, gestures and haptic (touch and force) feedback.
• Help and feedback components: tool tips, progress indicators, messages and online help.

A good interface is consistent, simple, gives quick feedback and suits the user's skill.

Animation languages

An animation language is used to describe an animation, that is the objects, their attributes and their movement over time, in textual or programmatic form. These languages fall into three groups.

1. Linear-list notations. The animation is a list of events, each with the start frame, end frame and action. Example: 42, 53, B, ROTATE, "PROP", 1, 30 means from frame 42 to 53 rotate object PROP about axis 1 by 30 degrees. It is simple but not flexible, since there are no loops or conditions.

2. General-purpose languages. Ordinary programming languages (C, Java) extended with animation routines or libraries (OpenGL, Java 3D). They give full control with variables, loops and procedures, and the animator needs to be a programmer.

3. Graphical languages. Animation is built visually with timelines, key-frame editors and node graphs (Flash, Maya), so no programming is needed. Scripting inside them (ActionScript, MEL) adds control.

Other specialised languages include ASAS (Actor Script Animation System) and MIRA, which add actors, scripts and time control. VRML/X3D describes 3-D animated scenes for the web, and SMIL is used for timed presentations.

Abstraction levels of programming in multimedia

Multimedia programming can be done at several levels of abstraction. Higher levels hide detail and are easier to use, while lower levels give more control and speed.

1. Multimedia application level (authoring tools and scripts): complete applications built using authoring systems, such as Director or Flash and their scripting. The programmer deals with scenes, time lines and objects.
2. Programming language level: a general language (C, C++, Java) with multimedia libraries or toolkits is used. This gives flexibility with moderate effort.
3. Multimedia operating system / API level (system software): services of the OS such as device drivers, scheduling and media APIs (DirectX, OpenGL, Java Media Framework) offer calls for playing audio and video, drawing and synchronisation.
4. Device driver and hardware level: the driver communicates directly with the hardware (sound card, graphics card, video capture). It is fast but hardware-specific.

Moving up, the level of abstraction rises, the programming effort falls and portability improves, but control and performance drop.

Applications of multimedia in media integration

Media integration means combining text, graphics, images, audio, video and animation into one digital product, so that they are presented together and in synchronisation. Its main applications are:

• Education and training: e-learning, interactive tutorials, simulations and virtual labs that mix narration, animation and video, which improves understanding and retention.
• Entertainment: films with computer graphics, video games and music videos combine all media.
• Digital libraries, encyclopaedias and museums: kiosks and CD/web encyclopaedias present text with images, sound and clips.
• Business and advertising: presentations, product demonstrations, interactive catalogues and online advertising.
• Information and news: web portals and online news mix text, photos and video streams.
• Video conferencing and collaboration: real-time audio, video and shared documents are integrated.
• Healthcare and engineering: medical imaging with reports, telemedicine, and CAD visualisation.
• Virtual reality and simulation: training pilots, surgeons and tourists in an immersive environment.

Digital image representation

A digital image is a two-dimensional function $f(x,y)$ sampled on a regular grid, where $x,y$ are spatial coordinates and $f$ gives the intensity or colour at that point. Each grid element is a pixel. Digitisation involves two steps:

• Sampling: choosing the grid of $M\times N$ points, which sets the spatial resolution.
• Quantisation: assigning each sample one of $L=2^k$ discrete levels, where $k$ is the number of bits per pixel (colour depth).

The image is stored as a matrix:

$f=\begin{bmatrix} f(0,0) & f(0,1) & \cdots \\ f(1,0) & f(1,1) & \cdots \\ \vdots & & \ddots \end{bmatrix}$

Types

• Binary image: 1 bit per pixel (0 black, 1 white).
• Gray-scale image: typically 8 bits per pixel (0 to 255).
• Colour image: 24 bits per pixel in RGB, with 8 bits for each of the red, green and blue components. Indexed images use a palette.

Storage requirement: $\text{bits}=M\times N\times k$.

Example. A $4\times4$ 3-bit gray image (levels 0 to 7):

$\begin{bmatrix}0&1&1&0\\1&5&6&1\\1&6&7&1\\0&1&1&0\end{bmatrix}$

It needs $4\times4\times3=48$ bits. A $640\times480$ 24-bit colour image needs $640\times480\times24=7{,}372{,}800$ bits, which is $921{,}600$ bytes, about 0.88 MB, uncompressed.

Multimedia application development life cycle (MADLC)

Developing a multimedia application follows a staged life cycle, with feedback between stages.

1. Analysis (requirements and planning). The target users, purpose, content, budget, schedule, hardware/software platform and delivery (CD, web, kiosk) are defined, and the feasibility is checked. Example: for an e-learning course on Nepali geography, find that students are in grades 8 to 10, will use low-end PCs and need offline CD delivery.

2. Design. The structure and look are decided: the storyboard, flowchart of navigation, screen layouts, media list (text, images, audio, video, animation), interaction design and technical architecture. Example: a storyboard of each lesson screen with a map, narration and a quiz button.

3. Development (production). Media assets are created or collected: text is written, images are scanned or drawn, audio is recorded and edited, video is shot and compressed, animation is made. They are assembled in an authoring tool (Director, Flash, Authorware) or by programming, and interactivity is scripted. Example: record narration for each district, make a map animation and build the menu.

4. Testing. Check content accuracy, navigation, links, media synchronisation, performance and compatibility on target machines; alpha testing by the team, beta testing by sample users. Example: test whether narration stays in step with highlighting on a 1 GB RAM PC.

5. Implementation (delivery and distribution). The final product is packaged (installers, CD/DVD mastering, web hosting), documentation and user training are provided, and copyright clearances are completed.

6. Maintenance and evaluation. User feedback is collected, bugs are fixed, content is updated and new versions are released, which restarts the cycle.

Throughout, a project team of a manager, content expert, graphic designer, audio/video specialist, programmer and tester works together.

Colour models

A colour model is a mathematical system for representing colours as tuples of numbers.

1. RGB (additive). Red, Green and Blue light are added. With 8 bits per channel, black is (0,0,0) and white is (255,255,255). It follows how displays emit light.

2. CMY / CMYK (subtractive). Cyan, Magenta and Yellow inks absorb light from white paper:

$C=1-R,\quad M=1-G,\quad Y=1-B$

K (black) is added in CMYK, because the mix of C, M and Y gives a muddy brown rather than real black, and black ink is cheaper.

3. HSV / HSB / HSL (perceptual). Colour is described by Hue (angle 0 to 360 degrees on the colour wheel), Saturation (purity) and Value, Brightness or Lightness. It matches how people think of colour and makes colour picking and editing easy.

4. YUV / $YC_bC_r$ / YIQ (luminance-chrominance). Y is the brightness, the other two are colour differences:

$Y=0.299R+0.587G+0.114B$

The eye is more sensitive to Y than to colour, so chrominance can be subsampled. YUV is used for PAL TV, YIQ for NTSC, and $YC_bC_r$ for JPEG and MPEG. Y alone gives a compatible black-and-white signal.

5. CIE XYZ / Lab. Device-independent models based on human vision; Lab is perceptually uniform, so equal distances look like equal colour differences.

Comparison

Abstraction levels of programming in a multimedia system

Programming a multimedia system is organised in layers. Each layer uses the services of the one below and hides its details from the one above.

   +----------------------------------------+
   | 5. Multimedia application              |  authoring tools, user programs
   +----------------------------------------+
   | 4. Scripting / authoring language      |  Lingo, ActionScript
   +----------------------------------------+
   | 3. Programming language + toolkits     |  C/C++, Java + libraries
   +----------------------------------------+
   | 2. System software (OS and API)        |  DirectX, OpenGL, JMF, media services
   +----------------------------------------+
   | 1. Device drivers and hardware         |  sound card, graphics, capture card
   +----------------------------------------+

• Level 1: device drivers and hardware. Hardware-specific code directly controls devices such as the sound card, video capture and graphics adapter. It has the highest speed and control, but is the least portable.
• Level 2: system software / operating system and API. Provides device-independent calls for media I/O, real-time scheduling, resource management and synchronisation, for example DirectX, OpenGL, Java Media Framework.
• Level 3: programming languages and toolkits. General languages with multimedia libraries or class frameworks allow flexible custom applications (object oriented support for media objects).
• Level 4: scripting and authoring tools. Higher-level tools using timelines, icons and scripts let designers combine media without detailed programming.
• Level 5: user-level multimedia applications. The end product, such as a tutorial, game or presentation, is used through the interface.

Moving upward gives ease of use, faster development and portability, but less control and efficiency. Moving downward gives more control and performance but needs more programming effort.

Challenges for multimedia systems

• Huge data volume: uncompressed audio and video need very large storage and bandwidth (for example, uncompressed video at 25 frames/s of $640\times480$ 24-bit pictures is about 184 Mbit/s). Efficient compression is therefore required.
• Real-time processing and delivery: continuous media must be processed and played on time, so the system needs real-time scheduling and fast processors.
• Synchronisation: audio, video, text and animation must be kept in step (for example lip sync), within limits of a few tens of milliseconds.
• Quality of service (QoS): networks must guarantee bandwidth, low delay, low jitter and low loss, which the best-effort Internet does not do.
• Storage and retrieval: large media files need fast, high-capacity storage, indexing and content-based search.
• Heterogeneity and standards: different formats, devices and platforms need standards and conversion for interoperability.
• Compression quality trade-off: lossy compression saves space but reduces quality.
• User interface design: it must be easy and intuitive for many kinds of users.
• Security and copyright: digital copies are perfect and easy to pirate, so watermarking and DRM are needed.
• Cost and content creation: producing good multimedia content needs skilled people and tools, and is expensive.

Speech generation in a multimedia system

Speech is generated by a text-to-speech (speech synthesis) system. The computer turns text or stored data into a sound waveform that is played through a speaker.

Approaches

1. Playback of stored speech. Complete words or phrases are recorded, digitised and compressed, then played in order. Quality is natural, but only a fixed vocabulary is possible and storage grows with it.
2. Concatenative synthesis. Short recorded units (phonemes, diphones, syllables) are joined to produce any word. The diphone, running from the centre of one phoneme to the centre of the next, preserves the transitions between sounds. Smoothing at the joins and pitch/duration adjustment are required.
3. Formant (rule-based) synthesis. A source-filter model is used. The source is a periodic pulse train for voiced sounds, or noise for unvoiced ones, and it passes through filters that imitate vocal tract resonances (formants). Rules control the formant frequencies over time. It needs little storage and has an unlimited vocabulary, but the voice is robotic.

Text-to-speech pipeline

1. Text analysis and normalisation (expand numbers, abbreviations).
2. Conversion of letters to phonemes (pronunciation dictionary and rules).
3. Prosody generation (pitch, duration, stress).
4. Waveform generation by one of the above methods.
5. Digital-to-analogue conversion and output.

Uses include screen readers, telephone information systems, talking books and navigation aids.

Image and graphics formats

A format defines how image data is stored in a file. It can be a bitmap (raster) format, a vector format, or both.

Bitmap formats

• BMP (Bitmap): Windows format, usually uncompressed, simple and large. Supports 1 to 24 bits per pixel.
• GIF (Graphics Interchange Format): up to 256 colours (8-bit palette), LZW lossless compression, supports transparency and simple animation. Suited to logos and icons.
• JPEG/JPG: lossy DCT-based compression for photographs, with 24-bit colour and adjustable quality. No transparency.
• PNG (Portable Network Graphics): lossless compression (Deflate), up to 48-bit colour and an alpha channel for transparency. Used on the web as a patent-free replacement for GIF.
• TIFF: flexible, high-quality format, normally lossless; used in publishing and scanning.

Vector and metafile formats

• SVG: XML-based scalable vector format for the web.
• EPS / PostScript, AI, CDR: printing and drawing application formats.
• WMF/EMF: Windows metafile formats storing drawing commands.
• CGM, DXF: engineering and CAD graphics interchange.

Methods of controlling animation

Controlling animation means specifying how objects move and change over time. The main methods are:

1. Explicit control (full explicit). The animator states the motion completely: transformations such as translation, rotation and scaling are given frame by frame, or key frames with interpolation are used. Key-frame control specifies only important poses and the computer fills in between (in-betweening) using linear or spline interpolation.
2. Procedural control. The motion is defined by a procedure or rules, with objects communicating, so that the movement of one object can depend on another. Examples are particle systems and flocking behaviour (boids).
3. Constraint-based control. Motion follows constraints, for example an arm attached to a body, or objects that cannot pass through each other. Inverse kinematics finds joint angles that place the end of a limb at a target.
4. Control by analysis of live action (motion capture). Real movements of an actor, recorded with sensors or cameras, are mapped to the model, which gives realistic motion. Rotoscoping traces over film frames.
5. Kinematic and dynamic control. Kinematic control describes motion by positions and velocities; dynamic control uses physical laws (forces, mass, gravity, collision) to calculate motion automatically.
6. Behavioural / actor-based control. Objects are actors with rules, goals and perception, which decide their own behaviour (for example in games and crowds).
7. Tracking live action / scripting. Animation languages or scripts specify timing and events.

Colour dithering

Dithering is a technique that makes an image with a limited number of colours (or gray levels) look as if it has more. It is used when the display or format supports few colours, for example a 256-colour palette or a 1-bit black-and-white printer.

Principle. The eye averages the colours of neighbouring small pixels. By placing pixels of the available colours close together in a pattern, the eye perceives an intermediate colour or shade. A region with 50% black and 50% white pixels in a fine pattern looks gray.

Common methods

• Ordered dithering (halftoning): the pixel value is compared with a threshold from a repeating dither matrix. For a $2\times2$ matrix $\begin{bmatrix}0&2\\3&1\end{bmatrix}$ scaled to the gray range, a block of four pixels can show 5 intensity levels (0 to 4 pixels on).
• Error diffusion (Floyd-Steinberg): each pixel is rounded to the nearest available colour, and the quantisation error is spread to neighbouring unprocessed pixels with weights $7/16$ (right), $3/16$ (below left), $5/16$ (below) and $1/16$ (below right). It gives better detail.
• Random dithering: random noise is added before thresholding. It avoids patterns but looks grainy.

Effects. Dithering reduces banding and false contours, improving apparent quality at the cost of some resolution and a grainy look. It is used in GIF images, early web graphics and printing.

JPEG and MPEG

Motion JPEG is a sequence of independent JPEG frames, and so is less efficient than MPEG, but it is easier to edit frame by frame.

Lossless and lossy compression

Design issues of an audio user interface

An audio user interface presents information and accepts commands through sound and speech. Its design has these issues.

• Speech recognition accuracy: accents, background noise, vocabulary size and speaker variation cause recognition errors, so the system needs confirmation and error recovery.
• Speech output quality: synthetic voice should be natural and clear, with correct pronunciation and prosody.
• Transient and linear nature of sound: audio is heard once and in sequence, unlike a screen where information stays visible. Users can forget it, so short messages, repetition and the ability to replay are needed.
• Navigation and orientation: without visual cues, users can get lost in menus, so menus should be short (few options), with clear prompts and a way to go back or ask for help.
• Sound cues: auditory icons and earcons (short musical tones) are used for feedback and alerts. They need to be distinct, meaningful and not annoying.
• Cognitive load and memory: the user must remember options, so the design should limit information in each prompt.
• Interruption (barge-in): the user should be able to interrupt long prompts.
• Environment and privacy: noise can mask the audio, and speaking aloud or loud output can disturb others or reveal private data.
• Accessibility and language: it should support visually impaired users, and multiple languages and dialects.
• Timing and response: delays in response and recognition frustrate users.

Multimedia in video conferencing

Video conferencing lets people at different places see and hear each other in real time, and often share documents and screens. It is a multimedia application which combines audio, video, text and data.

Working. At each site a camera and microphone capture video and audio, which are digitised and compressed by codecs (H.261/H.263/H.264 for video, G.711/G.723/G.729 for audio). The data is packetised and sent over the network (ISDN, LAN/Internet using RTP/UDP, with H.323 or SIP for signalling). At the receiver it is decoded, synchronised and played on a monitor and loudspeaker. For multi-party calls a multipoint control unit (MCU) mixes the streams.

Multimedia features used

• Real-time audio and video with synchronisation (lip sync).
• Whiteboard, text chat, file transfer and application/screen sharing.
• Compression to fit the available bandwidth, and echo cancellation for audio.
• QoS control for low delay and jitter.

Applications

• Business meetings and distance collaboration, saving travel cost.
• Distance education and virtual classrooms.
• Telemedicine and remote consultation.
• Legal proceedings, interviews and customer support.
• Family and personal communication (Skype, Zoom, Google Meet).

Challenges: bandwidth limits, delay, packet loss, equipment cost and security.

Why speech synthesis is difficult

Speech synthesis (text-to-speech, TTS) converts written text into an audible speech signal. Natural human speech carries much more information than the letters of the text, so producing it is hard.

• Text normalisation and ambiguity. Numbers, dates, abbreviations and symbols must be expanded ("Rs 5,000", "Dr.", "2082/05/12"). The same spelling can be read differently: "read" (present/past), "lead", "bass".
• Irregular letter-to-sound rules. In English the spelling does not give the pronunciation (cough, though, through). Exceptions, names and loan words need a dictionary as well as rules.
• Prosody. Pitch (intonation), duration (rhythm) and loudness (stress) are not written in the text. "You are going?" and "You are going." differ only in prosody. Wrong prosody makes speech sound robotic.
• Co-articulation. The sound of a phoneme changes with its neighbours, so joining isolated phonemes gives audible discontinuities.
• Speaker variation and emotion. Age, gender, accent, mood and speaking rate change the signal. Emotion and expressiveness are hard to model.
• Context and meaning. Correct stress and phrasing need syntactic and semantic understanding of the sentence.
• Real-time and storage limits. A good system must run fast and with moderate memory.
• Evaluation. Naturalness and intelligibility are subjective and are measured by listening tests.

Techniques for speech generation

A TTS system has a front end (text analysis: normalisation, grapheme-to-phoneme conversion, prosody prediction) and a back end (waveform generation). The back-end techniques are:

1. Time-dependent sound concatenation (concatenative synthesis). Recorded units (phonemes, diphones, syllables, words) are stored and joined. Diphones (the transition from the middle of one phone to the middle of the next) capture co-articulation, so about 1,500 to 2,000 diphones cover a language. Pitch and duration are adjusted with methods such as PSOLA. It sounds natural but needs a large recorded database and has joining artefacts. Unit-selection synthesis picks the best long units from a big corpus.
2. Frequency-dependent (formant / rule-based) synthesis. Speech is produced by a source-filter model: a source (periodic pulse train for voiced, noise for unvoiced sounds) is passed through resonance filters that model the vocal tract formants. Rules supply the formant frequencies, bandwidths and transitions for each phoneme. It needs little memory and is flexible, but sounds artificial.
3. Articulatory synthesis. Models the physical vocal organs (tongue, lips, glottis); accurate in theory but computationally heavy.
4. Statistical parametric synthesis (HMM) and neural synthesis. Models learn the acoustic parameters from data and a vocoder generates the waveform. Modern neural systems (Tacotron, WaveNet) give very natural voices.

The generated digital signal goes through a D/A converter and amplifier to the speaker.

Methods of controlling animation

• Frame based: the animation is a stored series of images (24-30 frames per second for smooth motion). It is simple to play back but needs a lot of storage.
• Key frame: the animator defines the start and end poses (position, size, colour, rotation); in-betweening interpolates linearly or along splines, so only the key frames are stored.
• Timeline: time is the main axis. Each sprite, sound or effect has a track with start and end times; the author can synchronise animation with audio and other media.

Transmission of animation

Animation can be sent over a network in two ways.

1. Symbolic (script/vector) transmission. Only the description is sent: the object definitions, the key frames and the animation commands (for example SWF, SVG animation, VRML, OpenGL display lists). The receiver renders the frames locally. This needs very little bandwidth and is resolution independent, but the receiver needs computing power and a compatible player.
2. Pixmap (rendered image) transmission. Each frame is rendered at the sender and sent as a bitmap sequence, usually compressed like video (MPEG, GIF). It needs high bandwidth (for example $640\times480\times24$ bits $\times 25$ fps is about 184 Mbit/s uncompressed), but the receiver only displays the frames.

Symbolic transmission is preferred for the web; pixmap transmission is used when the receiver is simple or the scene is complex. Synchronisation with audio and a steady frame rate (using buffering and streaming) are needed in both.

Entropy coding

Entropy coding is a lossless compression technique that ignores the meaning of the data and treats it as a plain sequence of bits. It uses the statistics of the data: frequent symbols get short codes and rare symbols get long codes. The average information per symbol is the entropy

$H = -\sum_{i} p_i \log_2 p_i \ \text{bits/symbol}$

and no lossless code can use fewer bits on average. Examples are run-length coding, Huffman coding, arithmetic coding and LZW.

Hybrid coding

Hybrid coding combines source coding (lossy, uses the data's semantics, such as prediction and transformation like DCT) with entropy coding (lossless). The source coding step removes redundancy and irrelevance, and entropy coding then removes the statistical redundancy. JPEG, H.261 and MPEG are hybrid coders: DCT, quantisation, then run-length and Huffman coding.

Lossy sequential DCT-based mode (JPEG baseline)

In this mode the image is encoded in a single left-to-right, top-to-bottom scan. The steps are:

1. Image preparation. Colour image components (for example RGB converted to YCbCr) are separated; the chrominance may be subsampled (4:2:0) because the eye is less sensitive to colour. Each component is divided into $8\times 8$ pixel blocks, and sample values are level shifted (subtract 128).
2. Forward DCT (FDCT). Each block is transformed from the spatial domain to the frequency domain:

$F(u,v)=\frac{1}{4}C(u)C(v)\sum_{x=0}^{7}\sum_{y=0}^{7} f(x,y)\cos\frac{(2x+1)u\pi}{16}\cos\frac{(2y+1)v\pi}{16}$

with $C(k)=1/\sqrt2$ for $k=0$ and $1$ otherwise. $F(0,0)$ is the DC coefficient (block average) and the other 63 are AC coefficients. DCT itself is nearly lossless; energy is concentrated in the low-frequency coefficients. 3. Quantisation. Each coefficient is divided by the corresponding entry of a quantisation table and rounded: $F_q(u,v)=\mathrm{round}\big(F(u,v)/Q(u,v)\big)$. High frequencies get large divisors, so many become zero. This is the only lossy step, and the table sets the quality/compression trade-off. 4. Entropy coding.

• The DC coefficient is coded as the difference from the previous block's DC (DPCM).
• The 63 AC coefficients are read in zig-zag order so zeros group together, then run-length coded as (run of zeros, value) pairs.
• The results are compressed with Huffman (or arithmetic) coding.

The decoder reverses the process: entropy decoding, dequantisation ($F_q\cdot Q$) and inverse DCT. The reconstructed image differs slightly from the original because of quantisation.

Multimedia system

A multimedia system is a computer-based system that can create, store, process, integrate, transmit and present information in more than one medium, such as text, graphics, images, audio, video and animation. At least one medium is continuous (time-dependent), such as audio or video, and the media are controlled and integrated by the computer.

Properties (characteristics)

• Combination of media. Both discrete (text, graphics, images) and continuous media (audio, video, animation) are used together.
• Computer controlled. The computer produces, manipulates, stores and presents the information.
• Integration. The media are linked in one system through a common interface, with a common representation and synchronisation (for example lip-sync between audio and video).
• Interactivity. The user can control what is presented and when (navigation, hyperlinks, menus).
• Digital representation. All media are stored as digital data, which makes processing, compression and transmission easy.
• Independence of media. Each medium can be handled separately, but they are presented as one.
• Real-time and time-dependent processing. Continuous media need timing guarantees, adequate bandwidth, low delay and jitter.
• Large storage and bandwidth demand. Compression and fast storage are needed.

Image synthesis

Image synthesis is the generation of an image from a description (a model of objects, light and camera) instead of capturing it with a camera. It is the work of computer graphics and is done in these phases:

1. Modelling (scene specification). The objects are described geometrically, using polygons, curves, surfaces or solids, with attributes such as colour, texture and material. Light sources and the camera (viewpoint) are also defined.
2. Geometric transformation. Objects are placed in the scene by translation, rotation and scaling, and then converted from world coordinates to viewing coordinates and projected (parallel or perspective) onto the view plane.
3. Clipping and hidden surface removal. Parts outside the view volume are cut away, and surfaces hidden behind other surfaces are removed (z-buffer, painter's algorithm, back-face culling).
4. Shading and rendering. The colour of each visible point is computed from the light sources, surface properties and viewing direction, using illumination models (ambient, diffuse, specular) with Gouraud or Phong shading, or ray tracing and radiosity for shadows, reflections and global illumination. Textures are applied.
5. Scan conversion (rasterisation). The geometric primitives are converted into pixels, and the colour values are written into the frame buffer.
6. Display. The frame buffer is read by the video controller and displayed on the screen (or stored as an image file).

Audio versus video as a user interface

Neither is better in every case. Video (visual) is generally the more effective primary interface, and audio is a strong complement or an alternative in particular situations.

Why video / visual is more effective

• Humans receive most of their information through vision, and the screen can show a lot at once (menus, icons, text, images), so the user can scan and choose quickly.
• Visual information is persistent: it stays on the screen so the user can reread it, while audio is sequential and temporary and must be remembered.
• It is silent, private, and works in noisy places.
• Pointing and direct manipulation (GUI) are natural and give instant feedback.

Where audio is more effective

• When the eyes or hands are busy (driving, operating machinery) or the user cannot look at a screen.
• For visually impaired users (screen readers, voice assistants).
• For alerts, warnings and notifications, which attract attention from any direction.
• Speech is a natural and fast input method for simple commands.
• It needs little display hardware, and works over a telephone.

Conclusion

Video is more effective for rich, detailed and reference information; audio is more effective for hands-free use, accessibility and alerts. The best multimedia interface combines both, so the user can use whichever suits the task and the environment.

System software abstractions for programming multimedia

Multimedia programming needs abstractions that hide device details and provide timing, synchronisation and media handling. The abstraction levels are:

1. Device (hardware) level. Direct access to hardware, for example a sound card or video capture card, through registers and device drivers. It is fast but hardware-specific and hard to program.
2. Operating-system / library level. Device drivers and system calls give uniform access to devices (audio, video, graphics, files), with real-time scheduling and resource management. APIs such as DirectX, OpenGL, ALSA and Java Sound belong here.
3. Basic multimedia object (toolkit/class) level. Libraries of objects such as sound, image, video clip or stream with operations (play, stop, record). Examples are Java Media Framework, QuickTime and GStreamer. Streams connect sources to sinks with filters in between.
4. Application / scripting (authoring) level. High-level tools and languages such as Director/Lingo, Flash ActionScript, SMIL and HTML5 media elements, where the author describes the presentation (timing, layout, interaction) without dealing with low-level details.
5. Specification/user level. Graphical authoring tools in which the user drags and links media on a timeline or flowchart.

Moving up the levels gives simplicity, portability and productivity, while moving down gives control and performance.

Video conferencing

Video conferencing is real-time, two-way communication in which participants at different locations see and hear each other over a network, using cameras, microphones, displays, speakers and a codec for compression (H.261/H.263/H.264). It may be point-to-point or multipoint (through a multipoint control unit, MCU). It can also share documents and screens. Examples are Zoom, Google Meet and Skype. It saves travel cost and time, but needs bandwidth and low delay.

Editing digital video

Video editing is done on a computer with non-linear editing software (Premiere, DaVinci Resolve, Kdenlive). The steps are:

1. Capture / import. Video is captured from a camera or imported as digital files, and clips are stored on disk.
2. Organise. Clips are logged and arranged in a project bin.
3. Assemble on a timeline. Clips are cut (trimmed), split, copied and arranged on tracks in any order, without altering the original files (non-destructive, random access).
4. Transitions. Cuts, fades, dissolves and wipes join the clips.
5. Effects and correction. Colour correction, filters, titles, overlays, speed change and compositing are added.
6. Audio editing. Narration, music and sound effects are added and synchronised with the picture and mixed.
7. Render and export. The edited timeline is rendered and compressed to the output format (MP4/H.264, MPEG) for the web, DVD or broadcast.

Characteristics of a multimedia system

• It combines several media (text, graphics, images, audio, video, animation), at least one of them time-dependent.
• It is computer controlled and uses a digital representation.
• The media are integrated and synchronised.
• It is interactive, so the user controls navigation.
• It needs high storage capacity and bandwidth, so compression is used.
• It needs real-time processing with low delay and jitter for continuous media.
• It has a common interface and is independent of the medium.

Representation of a digital image

A digital image is a two-dimensional array of discrete picture elements (pixels). Each pixel at position $(x, y)$ holds an intensity or colour value.

• Sampling divides the image into an $M\times N$ grid. The resolution is the number of pixels (for example $1024\times768$).
• Quantisation assigns each sample a value with a fixed number of bits (bit depth). A bit depth of $b$ gives $2^b$ levels: 1 bit gives black and white, 8 bits give 256 grey levels, and 24 bits give about 16.7 million colours.
• Colour is stored as three components per pixel (RGB, 8 bits each) or through a palette (indexed colour).

The storage needed is

$\text{size} = M\times N\times b \ \text{bits}$

For example, a $1024\times768$, 24-bit image needs $1024\times768\times24/8 = 2{,}359{,}296$ bytes, about 2.25 MB uncompressed. Images are stored as bitmaps (raster: BMP, JPEG, PNG, GIF) or described by vectors (SVG).

Why compress multimedia data

Multimedia data, especially audio and video, is very large, and uncompressed it needs too much storage and bandwidth.

• Uncompressed CD-quality stereo audio needs $44100\times16\times2 = 1.41$ Mbit/s.
• Uncompressed video of $640\times480$ pixels, 24 bits, 25 fps needs $640\times480\times24\times25\approx 184$ Mbit/s, about 23 MB per second or 1.4 GB per minute.
• Compression reduces storage cost, transmission time and bandwidth, and makes real-time streaming, video conferencing and web delivery possible over limited networks. It also lets disks and CD/DVD media hold more data.
• Compression is possible because the data contains redundancy (spatial, temporal, statistical) and irrelevance (details the eye or ear cannot notice).

Roles of the user interface in a multimedia system

• It is the link between the user and the system, and determines how easily the content can be accessed and used.
• It provides navigation and control (menus, buttons, hyperlinks, play/pause/seek) over interactive content.
• It integrates different media into one consistent presentation and gives feedback.
• It hides complexity: the user need not know the formats, devices and processing underneath.
• It supports accessibility through alternative inputs and outputs such as speech, captions and touch.
• A good UI gives a better learning experience, higher productivity and fewer errors.

Media integration

Media integration means combining different media (text, graphics, audio, video, animation) into one coherent system, so that they are handled and presented together under a common control, with a common representation and with temporal and spatial synchronisation. For example, a lecture video, its slides and its captions are shown in step with each other.

Expressing media as types, files and processes

• As types (data types). A medium is treated as an abstract data type with its attributes and operations, for example class Image (width, height, depth; show, scale), Audio (rate, samples; play, record) and Video (frame rate; play, seek). Types give a uniform way to program and combine media, and allow type checking and inheritance.
• As files. Media are kept as files in standard formats (JPEG, GIF, WAV, MP3, MPEG, AVI), and the system accesses them through the file system. Metadata in the header describes the encoding, size and rate. This is simple and portable, but continuous media need streaming, with timing and large sizes handled by the file system.
• As processes. A continuous medium is considered an active process or stream: a source process (camera, file reader) produces data, filter processes transform it (compress, mix) and a sink process (display, speaker) consumes it, connected through pipes or ports. This model fits real-time flow, with synchronisation between the processes.

Issues in designing a user interface (UI)

• Usability and consistency. The interface should be easy to learn and use, with consistent layout, terms and behaviour.
• User diversity. Users differ in age, skills, language, culture and disability, so the design should suit the target audience and offer accessibility.
• Choice of media. The right media (text, audio, video) must be chosen for the content without overloading the user (cognitive load).
• Navigation. Users should not get lost, and need clear menus, links, back and home options.
• Feedback and response time. The system should respond quickly and clearly to actions.
• Presentation. Layout, colour, font, resolution and screen size must be suitable, including for different devices.
• Synchronisation and performance. Media must be played in step without delay on the available bandwidth and hardware.
• Error handling and help. Messages must be clear and recovery easy.
• Platform independence and cost. Cross-platform use and development cost must be balanced.

Role of multimedia in telemedicine

Telemedicine provides medical care at a distance, and multimedia makes it effective.

• Video conferencing allows consultation between a patient in a remote village and a specialist in a city hospital.
• Medical images (X-ray, CT, MRI, ultrasound) are digitised, compressed and sent for diagnosis (teleradiology).
• Audio allows remote listening to heart and lung sounds (electronic stethoscope) and voice consultation.
• Records and data (text, ECG graphs, patient history) are shared as multimedia electronic records.
• Remote monitoring and education include live surgery demonstrations, tele-surgery support and training of health workers.

It saves travel, brings expert care to rural areas such as those in Nepal, and speeds up emergency treatment, but it needs reliable bandwidth, high-quality image transmission and data privacy.