Multimedia Computing

Time-dependent vs frequency-dependent sound concatenation

Sound concatenation joins stored sound segments to form speech output.

Components of a speech transmission system

1. Speech input and analysis – a microphone captures the signal, which is sampled and digitized (A/D conversion).
2. Encoding/compression – the digital signal is compressed using coders such as PCM, ADPCM or LPC to reduce bandwidth.
3. Transmission channel – the encoded data is sent over the network or telephone line, with error control.
4. Decoding – the receiver decompresses the data.
5. Speech synthesis / reproduction – D/A conversion, filtering and an amplifier and loudspeaker produce audible sound.

Elements of speech output include a text analyser (text to phonemes), prosody generator and a signal-synthesis stage.

Dithering

Dithering is a technique for creating the illusion of more colours or grey levels than the display can actually show. Pixels of the few available colours are arranged in patterns so that the eye blends them. For example, a black-and-white device can show greys by varying the density of black dots. Methods include ordered dither (threshold matrices), random dither and error diffusion (Floyd-Steinberg). It is used when reducing a true-colour image to a limited palette.

Image analysis

Image analysis extracts meaningful information from a digital image. It generally involves:

1. Image acquisition and digitization.
2. Pre-processing – noise removal, contrast enhancement, filtering.
3. Segmentation – dividing the image into regions or objects using thresholding, edge detection (gradient operators) or region growing.
4. Feature extraction – measuring shape, size, colour, texture, and boundaries.
5. Analysis and interpretation – measuring distances, areas, counts, and tracking motion.

Steps of image recognition

1. Image acquisition – capture the image with a camera or scanner.
2. Pre-processing – remove noise, normalize size and brightness.
3. Segmentation – separate objects from background.
4. Feature extraction – compute descriptors (edges, corners, shape, colour, moments).
5. Classification / matching – compare features with stored models or use a trained classifier (template matching, statistical or neural network) to identify the object.
6. Decision and output – recognised class is reported, with a confidence value, and checked against knowledge or context.

Temporal aspects of illumination

A video image is a sequence of still frames. To show motion without flicker, frames are displayed fast enough to exploit the persistence of vision of the eye.

• The frame rate must be sufficiently high (about 25-30 frames/s for motion), but the eye perceives flicker below about 50 Hz refresh.
• Interlacing is used: each frame is split into two fields (odd and even lines), so the screen is refreshed 50 times (PAL) or 60 times (NTSC) per second while only 25 or 30 full frames per second are sent, saving bandwidth.
• Phosphor persistence of the screen and display refresh rate affect perceived flicker and smoothness. Illumination of the scene also changes over time and the camera must track this (exposure, shutter).

Video bandwidth

Bandwidth is the range of frequencies required to carry the video signal. It depends on the number of picture elements per second: $B \approx \frac{\text{lines}\times \text{pixels per line}\times \text{frame rate}}{2}$ For example, $625$ lines, $\approx 833$ pixels per line (4:3 aspect), $25$ frames/s gives about $13\ \text{MHz}$ before interlacing allowance, and in practice about $5$ to $6$ MHz for a PAL channel after accounting for interlace and the Kell factor. Higher resolution and frame rates need more bandwidth, which is why compression is used.

Computer video controller standards

• MDA – monochrome text only, $720\times350$.
• CGA – $320\times200$ with 4 colours, $640\times200$ monochrome.
• EGA – $640\times350$ with 16 colours.
• VGA – $640\times480$ with 16 colours, $320\times200$ with 256 colours; analog output.
• SVGA – $800\times600$ and above with up to 16 million colours.
• XGA – $1024\times768$; later standards such as SXGA ($1280\times1024$), UXGA ($1600\times1200$), HD and Full HD.

A multimedia system is a computer-based system that captures, stores, processes, integrates and presents multiple media types (text, graphics, images, audio, video and animation) in an interactive, digitally controlled way.

Components

1. Capture devices: keyboard, mouse, scanner, digital camera, microphone, video camera and graphics tablet used to input media.
2. Storage devices: hard disk, SSD, DVD/Blu-ray, flash memory and cloud storage, since audio and video need large capacity.
3. Processing hardware: a fast CPU, large RAM, GPU, sound card and video card for encoding, decoding and rendering.
4. Communication networks: LAN, Internet and broadcast links needed for streaming and distributed multimedia, with enough bandwidth and low delay.
5. Presentation (output) devices: monitor, projector, speakers, headphones and printers.
6. Software: operating system with multimedia support, authoring tools (e.g. Flash, Director), editing tools (Photoshop, Audacity, Premiere), codecs and multimedia databases.
7. User interface: lets the user control navigation and interaction (hypermedia links, menus, buttons).

MIDI (Musical Instrument Digital Interface) is a standard protocol and hardware interface that lets electronic musical instruments, computers and sound modules exchange music control information (notes, velocity, pitch, instrument selection) rather than the sampled sound itself. A MIDI file therefore stores events, so it is very compact.

Components of a MIDI interface

• MIDI ports: three 5-pin DIN connectors. IN receives data, OUT sends data generated by the device, and THRU passes on a copy of the received data to the next device in a chain.
• MIDI cable: a shielded twisted-pair cable carrying serial data at 31.25 kbaud, which is asynchronous with 1 start bit, 8 data bits and 1 stop bit.
• Controller/keyboard (MIDI instrument): generates messages such as Note On, Note Off, pitch bend and program change.
• Sequencer: software or hardware that records, edits and plays back MIDI messages.
• Synthesizer/sound module: converts MIDI messages into sound using wavetable or FM synthesis.
• Channels: 16 channels let several instruments share one link.

Computer-based animation creates the illusion of motion by displaying a sequence of images (frames) quickly, usually 24-30 frames per second.

Steps

1. Storyboarding / concept: the story is planned as a series of sketches giving scenes, characters and actions.
2. Object definition: the objects and characters are modelled (2D shapes or 3D models), and their motion, colours and textures are specified.
3. Key-frame specification: the animator defines the key frames, the important poses at the start and end of each action.
4. In-betweening (tweening): the computer generates the intermediate frames between key frames by interpolation.
5. Motion specification: paths, speed, acceleration, rotation and scaling are defined, using methods such as key-frame, procedural or physical simulation.
6. Rendering: lighting, shading, textures and camera view are applied to produce the final images.
7. Editing and post-processing: frames are sequenced, sound is synchronised and effects added.
8. Output and playback: the result is stored on film, video or in a digital format (GIF, MP4, SWF) and displayed at a fixed frame rate.

Multimedia programming is organised in layers of abstraction, from low-level hardware control to high-level authoring.

1. Device level (lowest): direct programming of hardware such as sound cards, video cards and capture cards through registers and device drivers. It is fast but hardware dependent and hard to use.
2. Operating system / driver level: the OS gives device-independent access through system calls and drivers, and manages resources, scheduling and real-time constraints.
3. Library / API level (toolkit): libraries such as DirectX, OpenGL, OpenAL and Java Media Framework give functions for playing audio, video, graphics and animation without handling hardware details.
4. Programming language level: general-purpose languages (C, C++, Java) or scripting languages (JavaScript, Lingo, ActionScript) use the APIs and give full control, including object-oriented multimedia classes.
5. Authoring system / tool level (highest): authoring tools (Director, Flash, Authorware, PowerPoint) let non-programmers build applications using timelines, icons or scripts with little or no code.

Moving up, ease of use and portability increase while flexibility and efficiency decrease.

Media preparation is the creation and conversion of the individual media elements before they are combined: text is typed and formatted, images are captured or drawn and edited, audio and video are recorded, digitised, edited and compressed, and animations are built. Each element is converted into a suitable file format, with sizes and quality adjusted for the target platform.

Media composition is the integration of these elements into one product. Using an authoring tool, the developer arranges elements on the screen, sets their timing and synchronisation (timeline or links), adds interactivity (buttons, navigation, hyperlinks), applies transitions, then tests and publishes the result.

Applications of multimedia

• Education and e-learning, computer-based training, simulations.
• Entertainment: games, movies, video on demand, music.
• Business: presentations, advertising, kiosks, video conferencing.
• Medicine: telemedicine, 3D anatomy, surgical training.
• Digital libraries, museums and tourism guides.
• Web design, virtual reality and engineering design.

Need for data compression

Uncompressed multimedia is huge. One second of CD-quality stereo audio is $44100 \times 16 \times 2 \approx 1.4$ Mbit, and raw video can need hundreds of Mbit/s. Compression is needed to:

• reduce storage requirements,
• reduce transmission time and bandwidth,
• allow real-time streaming and playback over limited networks,
• cut cost.

Lossy vs lossless compression

A multimedia user interface should make a complex mix of media easy and natural to use. The main design issues are:

1. Consistency: the same layout, icons, colours and commands across screens.
2. Navigation: clear paths, menus, links and a way to go back or home, so users do not get lost.
3. Media selection and balance: each medium used for a purpose (not overloaded), and text, graphics, sound and video integrated with synchronisation.
4. Screen layout and visual design: readable fonts, good contrast, colour use, white space and a clear focus.
5. Interactivity and feedback: immediate response to user actions, with progress indicators.
6. User control: pause, stop, rewind, volume and skip controls for time-based media.
7. Cognitive load: avoiding information overload, with simple and clear presentation.
8. Accessibility and diversity: support for disabled users (captions, alt text), languages and different skill levels.
9. Performance: acceptable delay and response time, and platform and bandwidth limits.
10. Usability testing: evaluation with real users and iteration.

Multimedia system development follows a life cycle with three main phases.

1. Analysis

• Identify the problem, goals, target audience and their needs.
• Study the content, available resources, budget, time, hardware and software constraints.
• Perform a feasibility study and prepare the requirement specification and project plan.

2. Design

• Define the structure and navigation (flowcharts, storyboards, site maps).
• Plan the content, media types, screen layouts, interface and interactivity.
• Choose authoring tools, file formats and compression, and write the design document and prototype.

3. Implementation

• Produce the media elements (text, graphics, audio, video, animation).
• Assemble and integrate them with an authoring tool or programming language, adding interactivity and synchronisation.
• Test (functional, usability, performance), fix errors, then deliver, document and train users.
• Maintain and update the product after release.