Developing multimedia applications with the

Java^TM Media Framework

( includes JMF 2.1.1 & MMAPI )

First Edition

T.G.Venkatesh

CONTENTS

Page

1. Introduction

1.1 Java for Internet computing ----------------------------------------------------------------- 15

1.2 Java Media APIs ---------------------------------------------------------------------------------- 16

1.3 Java Media Framework ----------------------------------------------------------------------- 18

What is JMF?------------------------------------------------------------------------------------ 18

Criticisms on JMF ---------------------------------------------------------------------------- 19

Why JMF? --------------------------------------------------------------------------------------- 19

1.4 Organization of the book --------------------------------------------------------------------- 20

2. Getting started with JMF

2.1 History and various releases of JMF ------------------------------------------------------ 25

2.2 Versions of JMF ----------------------------------------------------------------------------------- 25

2.3 Hardware platform requirements ----------------------------------------------------------- 26

2.4 Software requirements -------------------------------------------------------------------------- 26

Operating system ------------------------------------------------------------------------------ 26

Java platform requirements (JDK) ------------------------------------------------------ 26

Optional browser requirements ---------------------------------------------------------- 26

Optional Integrated development environments -------------------------------- 27

2.5 Media formats supported by the JMF ----------------------------------------------------- 27

2.6 Installing JMF ---------------------------------------------------------------------------------------- 27

2.7 Using multimedia devices with JMF ------------------------------------------------------- 27

Multimedia devices supported by JMF -------------------------------------- 27

Adding new devices --------------------------------------------------------------------------- 28

2.8 The JMF Registry ----------------------------------------------------------------------------------- 28

2.9 The JMStudio ---------------------------------------------------------------------------------------- 29

2.10 HelloJMF.java --------------------------------------------------------------------------------------- 34

2.11 Debugging JMF programs: the jmf.log file -------------------------------------------- 36

3. Multimedia : an introduction

3.1 General media compression techniques ----------------------------------------------- 39

Quantization ------------------------------------------------------------------------------------ 40

Run length coding --------------------------------------------------------------------------- 40

Entropy coding -------------------------------------------------------------------------------- 40

Predictive coding ---------------------------------------------------------------------------- 40

Transform coding --------------------------------------------------------------------------- 40

Discrete Cosine Transformation ------------------------------------------------------- 41

Zigzag scanning of DCT coefficients ----------------------------------------------- 41

Motion estimation --------------------------------------------------------------------------- 41

3.2 Audio coding techniques and standards ---------------------------------------------- 41

Speech codecs -------------------------------------------------------------------------------- 42

Vocoders --------------------------------------------------------------------------------------- 42

Waveform codecs -------------------------------------------------------------------------- 42

Pulse code modulation ------------------------------------------------------------------ 42

DPCM --------------------------------------------------------------------------------------------- 43

ADPCM ------------------------------------------------------------------------------------------ 43

Sub-band coding --------------------------------------------------------------------------- 43

Hybrid codecs -------------------------------------------------------------------------------- 43

Code Excited Linear Prediction ---------------------------------------------------- 44

3.3 Image coding standard ----------------------------------------------------------------------- 44

Color formats ---------------------------------------------------------------------------------- 44

JPEG format ------------------------------------------------------------------------------------ 45

3.4 Video coding standards ----------------------------------------------------------------------- 45

The CIF and QCIF standard ------------------------------------------------------------- 45

H261 standard -------------------------------------------------------------------------------- 45

H.263 videoconferencing standard for POTS --------------------------------- 46

Moving Picture Expert Group standards ----------------------------------------- 46

( MPEG-1, MPEG-2, MPEG-4, MPEG audio standard)

3.5 Proprietary standards ------------------------------------------------------------------------- 47

4 A quick tour to JMF

4.1 The JMF Model ------- ----------------------------------------------------------------------------- 53

4.2 Representing media ------------------------------------------------------------------------------ 53

4.3 Media source ----------------------------------------------------------------------------------------- 53

MediaLocator ----------------------------------------------------------------------------------- 54

DataSource ------------------------------------------------------------------------------------- 54

DataSink ----------------------------------------------------------------------------------------- 54

4.4 Media Playback ------------------------------------------------------------------------------------- 55

MediaHandler ---------------------------------------------------------------------------------- 55

Controller ----------------------------------------------------------------------------------------- 55

Player ---------------------------------------------------------------------------------------------- 55

4.5 Media Processing ---------------------------------------------------------------------------------- 55

PlugIn ---------------------------------------------------------------------------------------------- 56

Processor ------------------------------------------------------------------------------------------ 56

4.6 Managers ----------------------------------------------------------------------------------------------- 57

Manager -------------------------------------------------------------------------------------------- 57

PlugInManager ---------------------------------------------------------------------------------- 57

CaptureDeviceManager --------------------------------------------------------------------- 57

PackageManager ------------------------------------------------------------------------------- 57

4.7 JMF Controls ------------------------------------------------------------------------------------------- 57

4.8 JMF RTP API ------------------------------------------------------------------------------------------- 57

RTP --------------------------------------------------------------------------------------------------- 58

RTCP ------------------------------------------------------------------------------------------------- 58

RTPSession --------------------------------------------------------------------------------------- 58

4.9 The JMF Time model -------------------------------------------------------------------------------- 58

Time ------------------------------------------------------------------------------------------------- 58

TimeBase ------------------------------------------------------------------------------------------ 59

Clock ------------------------------------------------------------------------------------------------- 59

Duration -------------------------------------------------------------------------------------------- 62

4.10 Media Format ------------------------------------------------------------------------------------------ 62

Format ---------------------------------------------------------------------------------------------- 62

AudioFormat -------------------------------------------------------------------------------------- 63

VideoFormat--------------------------------------------------------------------------------------- 66

Buffer ------------------------------------------------------------------------------------------------ 66

ContentDescriptor ------------------------------------------------------------------------------- 66

The RGB Format --------------------------------------------------------------------------------- 68

The YUV Format ---------------------------------------------------------------------------------- 68

5 Media source

5.1 Specifying media source using MediaLocator ------------------------------------ 70

5.2 Specifying media source using URL ----------------------------------------------------- 74

5.3 DataSource ----------------------------------------------------------------------------------------- 76

5.4 DataSource interfaces ------------------------------------------------------------------------- 83

Positionable------------------------------------------------------------------------------------ - 83

SourceCloneable ----------------------------------------------------------------------------- 83

CaptureDevice --------------------------------------------------------------------------------- 85

RateConfigurable ----------------------------------------------------------------------------- 85

5.5 Push and Pull DataSources ------------------------------------------------------------------- 86

Buffer ---------------------------------------------------------------------------------------------- 87

DataSource Classification --------------------------------------------------------------- 87

5.6 SourceStreamClassification ------------------------------------------------------------------ 89

PullSourceStream --------------------------------------------------------------------------- 89

PushSourceStream ------------------------------------------------------------------------- 89

Pull and Push BufferStreams ----------------------------------------------------------- 92

6. Media presentation

6.1 The Controller interface ----------------------------------------------------------------------- 95

6.2 Controller states ----------------------------------------------------------------------------------- 97

Unrealized, Realizing and Realized states --------------------------------------- 97

Prefetching and Prefetched state ----------------------------------------------------- 97

Starting and stopping a Controller --------------------------------------------------- 98

Deallocating and closing a Controller ---------------------------------------------- 98

Startup latency --------------------------------------------------------------------------------- 98

6.3 The Player interface ----------------------------------------------------------------------------- 99

6.4 ControllerListener -------------------------------------------------------------------------------- 104

6.5 An experiment with the asynchronous Controller methods ------------------- 107

6.6 ControllerAdaptor --------------------------------------------------------------------------------- 110

6.7 Anatomy of a Player ----------------------------------------------------------------------------- 112

6.8 Controller Events ---------------------------------------------------------------------------------- 115

7. Media Capture

7.1 Supported capture devices ------------------------------------------------------------------ 124

7.2 Registering capture devices ------------------------------------------------------------------ 124

7.3 JMF API support for capturing media ---------------------------------------------------- 126

CaptureDevice interface -------------------------------------------------------------------- 126

Class CaptureDeviceInfo ------------------------------------------------------------------- 126

Class CaptureDeviceManager ----------------------------------------------------------- 128

7.4 Example to find registered capture devices ----------------------------------------- 128

7.5 Capturing media and presenting it to the user ------------------------------------ 130

Example for capturing media ---------------------------------------------------------- 131

Example for captured media ---------------------------------------------------------- 133

8 Media processing

8.1 Media processing stages ------------------------------------------------------------------------ 137

8.2 PlugInManager ------------------------------------------------------------------------------------- 140

PlugInManager methods ------------------------------------------------------------------ 140

Example to list registered PlugIns --------------------------------------------------- 142

8.3 ProccesorModel ------------------------------------------------------------------------------------- 143

8.4 Controlling the media processing --------------------------------------------------------- 147

Configured state ------------------------------------------------------------------------------ 149

Track ---------------------------------------------------------------------------------------------- 150

TrackControl ----------------------------------------------------------------------------------- 150

Processor’s output -------------------------------------------------------------------------- 151

Buffer----------------------------------------------------------------------------------------------- 152

ContentDescriptor -------------------------------------------------------------------------- 152

9 Processor configuration

9.1 The asynchronous nature of the configure() method. ---------------------------- 155

9.2 Using the configureCompleteEvent handler ------------------------------------------ 160

9.3 Invoking realize() in ConfigureCompleteEvent handler ------------------------- 163

9.4 Making Processor state transitions synchronous ------------------------------------ 165

9.5 Waiting for the Processor to reach the target state -------------------------------- 168

9.6 Setting codecs of a Processor ---------------------------------------------------------------- 174

9.7 Setting Renderers of a Processor ---------------------------------------------------------- 177

9.8 The utility ProcUtil.java ------------------------------------------------------------------------- 179

10 Media storage

10.1 DataSink ------------------------------------------------------------------------------------------ 188

10.2 DataSinkEvents & DataSinkListeners -------------------------------------------------- 190

10.3 Storing captured media into a file ----------------------------------------------------- 190

10.4 Storing both audio and video in the same file ----------------------------------- 194

11 Simple media transmission and reception

11.1 Media transmitting and receiving using JMStudio ------------------------------ 199

11.2 Media transmission using a DataSink ------------------------------------------------- 201

11.3 Media reception using a MediaLocator ----------------------------------------------- 206

12 JMF Applet

12.1 JMF Player in an Applet ------------------------------------------------------------------ 210

12.2 Media capture and storage in an applet ----------------------------------------- 213

12.3 Browser support for JMF----------------------------------------------------------------- 213

ADVANCED TOPICS

13 JMF RTP API

13.1 Real-time Transport Protocol -------------------------------------------------------------- 215

13.2 RTP Specification ----------------------------------------------------------------------------- 216

13.3 RTP Control Protocol ------------------------------------------------------------------------ 217

13.4 The JMF RTP API Terminology ---------------------------------------------------------- 218

13.5 The javax.media.rtp package ------------------------------------------------------------ 220

13.6 The package javax.media.rtp.rtcp ----------------------------------------------------- 223

13.7 The package javax.media.rtp.events ------------------------------------------------- 224

13.8 RTPEvent Listeners ---------------------------------------------------------------------------- 228

13.9 RTPEvent listener adaptors ---------------------------------------------------------------- 229

14 Multimedia communication using JMF RTP API- Part 1

14.1 Media reception using JMF RTP API -------------------------------------------------- 234

Media reception through a Processor ---------------------------------------------- 245

A GUI for media reception -------------------------------------------------------------- 245

14.2 Media transmission using JMF RTP API ---------------------------------------------- 248

Transmitting a media stream ---------------------------------------------- 249

Creating a media stream ---------------------------------------------------------------- 253

A GUI for media transmission ---------------------------------------------------------- 256

15. Multimedia communication using JMF RTP API- Part 2

15.1 Simultaneous transmission and reception of media ----------------------------- 260

15.2 Transfer of audio and video ------------------------------------------------------------------ 264

Audio and video transmission ---------------------------------------------------------- 264

Audio and video reception --------------------------------------------------------------- 271

15.3 Processing the received media. ------------------------------------------------------------- 273

Storing the received media -------------------------------------------------------------- 274

Retransmission of received media ---------------------------------------------------- 278

16 JMF Control

16.1 The interface Control and Controls ------------------------------------------------------ 281

16.2 Gain, mute, and rate control ----------------------------------------------------- 284

16.3 Buffer control -------------------------------------------------------------------------------------- 291

16.4 Quality control ------------------------------------------------------------------------------------ 295

16.5 Stream writer control ------------------------------------------------------------------------- 298

16.6 Monitor control ----------------------------------------------------------------------------------- 298

16.7 Bit rate control ---------------------------------------------------------------------------------- 300

16.8 Frame controls ---------------------------------------------------------------------------------- 301

17 Multiple DataSources and Players

17.1 Cloning DataSources --------------------------------------------------------------------------- 302

17.2 Merging DataSources --------------------------------------------------------------------------- 312

17.3 Controlling multiple players ----------------------------------------------------------------- 318

18 Multimedia Player Bean

18.1 JavaBeans component architecture ------------------------------------------ 323

18.2 Using MediaPlayer in JMF programs ---------------------------------------------------- 325

19 Extending JMF

19.1 Extending PlugIns ------------------------------------------------------------------------------ 329

19.2 Using PlugIn as a Processing stage ---------------------------------------------------- 335

19.3 Registering a PlugIn with the PlugIn Manager ----------------------------------- 339

19.4 An audio Effect PlugIn ----------------------------------------------------------------------- 342

19.5 A video Effect PlugIn -------------------------------------------------------------------------- 346

19.6 Defining new control interfaces for a PlugIn -------------------------------------- 351

19.7 Extending Multiplexer,Demultiplexer and Renderer -------------------------- 357

19.8 Extending DataSource, DataSink, and Controller -------------------------------- 358

19.9 Extentions related to JMF RTP API ---------------------------------------------------- 358

Extending Packetizer and DePacketizers ------------------------------------ 358

Dynamic payload ------------------------------------------------------------------------- 359

RTPConnector ------------------------------------------------------------------------------ 360

JMF-based Multimedia Projects

20. Multimedia Projects

Aim ----------------------------------------------------------------------------------------------------- 362

Design specifications -------------------------------------------------------------------------- 362

21. Video phone ----------------------------------------------------------------------------------------- 364

21.1 Detailed design specifications ------------------------------------------------------------ 366

21.2 Design of videophone ------------------------------------------------------------------------ 371

21.3 Implementation of videophone ---------------------------------------------------------- 371

User interface module --------------------------------------------------------------------- 374

Connection manager --------------------------------------------------------------------- 374

Videophone module ----------------------------------------------------------------------- 382

Chat manager --------------------------------------------------------------------------------- 385

Overall co-ordination ----------------------------------------------------------------------- 388

21.4 Source code of JMF classes ----------------------------------------------------------------- 390

22. Video & Voice mailing system

22.1 Detailed design specifications -------------------------------------------------------------- 391

22.2 Design of video & voice mailing system ------------------------------------------------ 391

22.3 Implementation of video & voice mailing system --------------------------------- 395

Mail Transmission module ---------------------------------------------------------------- 396

Mail Reception module -------------------------------------------------------------------- 399

Mail box module --------------------------------------------------------------- 401

Mail management module --------------------------------------------------- 403

22.4 Source code of JMF classes ---------------------------------------------------------------- 404

23. A simple Video Conferencing system

23.1 Design of a simple video conferencing system -------------------------------------- 405

23.2 Implementation of the video conferencing system -------------------------------- 406

23.3 Source Code of JMF classes ------------------------------------------------------------------ 409

24. Cosmophone

24.1 Integration of subsystems ------------------------------------------------------------------- 410

24.2 The help facility ---------------------------------------------------------------------------------- 410

24.3 Debugging ------------------------------------------------------------------------------------------ 411

24.4 Auto detection of capture devices ------------------------------------------------------ 411

24.5 Serial number ------------------------------------------------------------------------------------ 411

24.6 Documentation ----------------------------------------------------------------------------------- 412

24.7 Deployment of Cosmophone ---------------------------------------------------------------- 412

Special topics of JMF

1 Mobile media API

The J2ME platform ----------------------------------------------------------------------------------- 414

The Connected Limited Device Configuration (CLDC) ------------------------------ 414

The Mobile Information Device Profile (MIDP)------------------------------------------ 415

The MIDlet ----------------------------------------------------------------------------------------------- 416

What is MMAPI? -------------------------------------------------------------------------------------- 416

The MMAPI architecture -------------------------------------------------------------------------- 417

The DataSource ------------------------------------------------------------------------------------- 418

The Player --------------------------------------------------------------------------------------------- 418

The Manager ------------------------------------------------------------------------------------------ 420

Supported Media formats and protocols ------------------------------------------------ 421

The Player Control --------------------------------------------------------------------------------- 422

Developing multimedia MIDlets --------------------------------------------------------------- 422

The Wireless ToolKit (WTK) --------------------------------------------------------------- 422

The MM Emulator ------------------------------------------------------------------------------- 425

Example multimedia MIDlets -------------------------------------------------------------------- 426

Playing mono tone ---------------------------------------------------------------------------- 426

Playing a tone sequence -------------------------------------------------------------------- 428

Playing a wav file with simple playback control ------------------------------- 430

Playing a MIDI file ------------------------------------------------------------------------------ 432

Capturing audio and playing back -------------------------------------------------- 434

Capturing audio and recording -------------------------------------------------------- 434

Developing a MPEG Player ------------------------------------------------------------ 435

2 IBM’s HotMedia ----------------------------------------------------------------------------------------- 446

3. JMF Customizer --------------------------------------------------------------------------------------- 447

4. Anatomy of the JMF --------------------------------------------------------------------------------- 450

APPENDIX

Appendix A : Brief introduction to the UML notation used in the book. ----------------- 454

Appendix B : The hardware and software requirements of the various ---------------- 457

versions of JMF

Appendix C : The various media formats and capture devices supported ------------- 460

by JMF

Appendix D : Various Errors and exceptions thrown by the JMF objects. ------------ 465

Appendix E : Java-enabled smartphones supporting MMAPI -------------------------------- 467

References

Index