xii Contents
13.11 Distribution of estimates................................................................................................336
13.11.1 Central Limit Theorem....................................................................................337
13.12 Hypothesis testing and confidence intervals..................................................................338
13.12.1 Hypothesis Testing ..........................................................................................339
13.12.2 Confidence Regions.........................................................................................342
13.13 Empirical methods for hypothesis testing .....................................................................345
13.14 Summary........................................................................................................................346
13.A Appendix .......................................................................................................................347
13.A.1 Proof of Cramer-Rao Inequality......................................................................347
Chapter 14 Estimation Methods: Part I.......................................................................................350
14.1 Introduction ...................................................................................................................350
14.2 Method of moments estimators .....................................................................................351
14.2.1 Basic Idea ........................................................................................................351
14.3 Least squares estimators................................................................................................355
14.3.1 Ordinary Least Squares ...................................................................................355
14.3.2 Goodness of LS Fits ........................................................................................362
14.3.3 Properties of the LS Estimator ........................................................................365
14.3.4 Computing the Linear LS Estimate .................................................................373
14.3.5 Weighted Least Squares...................................................................................376
14.3.6 Other Variants of Linear LS ............................................................................381
14.4 Non-linear least squares ................................................................................................382
14.4.1 Numerical Methods for Optimization .............................................................384
14.4.2 Special Cases...................................................................................................386
14.4.2.1 Linear in Parameters.......................................................................387
14.4.2.2 Linear via Transformation..............................................................387
14.4.2.3 Pseudo-Linear Regression..............................................................388
14.4.2.4 Algorithmic Aspects of NLS Methods...........................................388
14.4.3 Asymptotic Properties of the NLS Estimator..................................................389
14.5 Summary........................................................................................................................393
14.A Appendix .......................................................................................................................394
14.A.1 Projection Theorem .........................................................................................394
14.A.2 Decomposition Theorem .................................................................................394
14.A.3 QR Factorization .............................................................................................395
14.A.4 Singular Value Decomposition........................................................................396
Chapter 15 Estimation Methods: Part II .....................................................................................400
15.1 Maximum likelihood estimators ...................................................................................400
15.1.1 Estimation of Mean and Variance: GWN........................................................403
15.1.2 Estimation of an ARX Model..........................................................................405
15.1.3 Computing the MLE........................................................................................409
15.1.4 Properties of the ML Estimator .......................................................................410
15.2 Bayesian estimators.......................................................................................................411
15.2.1 Linear Bayesian MMSE ..................................................................................416
15.3 Summary........................................................................................................................417
Contents xiii
Chapter 16 Estimation of Signal Properties................................................................................419
16.1 Introduction ...................................................................................................................419
16.2 Estimation of mean and variance ..................................................................................419
16.2.1 Estimators of Mean..........................................................................................420
16.2.2 Estimation of Variance ....................................................................................422
16.3 Estimators of correlation ...............................................................................................424
16.3.1 Estimators of Partial Correlation.....................................................................425
16.4 Estimation of correlation functions ...............................................................................426
16.5 Estimation of auto-power Spectra .................................................................................433
16.5.1 Periodogram ....................................................................................................434
16.5.2 Finite-Length Effects in Direct DFT Methods ................................................434
16.5.2.1 Spectral Leakage ............................................................................434
16.5.3 Remedies: Window Functions.........................................................................438
16.5.4 Estimation of Spectra for Stochastic Signals...................................................445
16.5.5 Periodogram Estimator....................................................................................445
16.5.5.1 Properties of Periodogram as a PSD Estimator for Stochastic
Signals ............................................................................................445
16.5.6 Averaged (Smoothed) Periodogram Estimators ..............................................451
16.5.7 Parametric Methods ........................................................................................461
16.5.8 Subspace Decomposition-Based Methods ......................................................464
16.6 Estimation of cross-spectral density..............................................................................466
16.7 Estimation of coherence ................................................................................................468
16.8 Summary........................................................................................................................473
PART IV IDENTIFICATION OF DYNAMIC MODELS - CONCEPTS AND
PRINCIPLES
Chapter 17 Non-Parametric and Parametric Models for Identification ......................................479
17.1 Introduction ...................................................................................................................479
17.2 The overall model..........................................................................................................479
17.3 Quasi-stationarity ..........................................................................................................480
17.4 Non-parametric descriptions ........................................................................................484
17.4.1 Time-Domain Descriptions .............................................................................484
17.4.1.1 FIR Models.....................................................................................485
17.4.1.2 Step Response Models ...................................................................485
17.4.2 Frequency-Domain Descriptions.....................................................................486
17.5 Parametric descriptions .................................................................................................486
17.5.1 Equation-Error Models....................................................................................488
17.5.1.1 ARX Family ...................................................................................488
17.5.1.2 ARMAX Family.............................................................................489
17.5.1.3 ARIMAX Models...........................................................................491
17.5.2 Output-Error Family........................................................................................492
17.5.3 Box-Jenkins Family.........................................................................................494
17.5.4 Selecting an Appropriate Model Structure......................................................496
17.6 Summary........................................................................................................................497
xiv Contents
Chapter 18 Predictions ...............................................................................................................499
18.1 Introduction ...................................................................................................................499
18.2 Conditional expectation and linear predictors...............................................................500
18.2.1 Best Linear Predictor.......................................................................................502
18.3 One-step ahead prediction and innovations...................................................................505
18.3.1 Predictions of the Stochastic Process ..............................................................505
18.3.2 Predictions of the Overall LTI System ............................................................506
18.4 Multi-step and infinite-step ahead predictions ..............................................................507
18.5 Predictor model: An alternative LTI description ...........................................................512
18.5.1 Model Sets and Structures...............................................................................513
18.6 Identifiability .................................................................................................................514
18.6.1 Model Identifiability........................................................................................514
18.6.2 Identifiable LTI Black-Box Structures ............................................................514
18.6.3 System Identifiability.......................................................................................517
18.7 Summary........................................................................................................................518
Chapter 19 Identification of Parametric Time-Series Models.....................................................520
19.1 Introduction ...................................................................................................................520
19.2 Estimation of AR models ..............................................................................................520
19.2.1 Y-W Method ....................................................................................................521
19.2.2 Least Squares / Covariance Method ................................................................524
19.2.3 Modified Covariance Method..........................................................................525
19.2.4 Burg’s Method .................................................................................................526
19.2.5 ML Estimator...................................................................................................529
19.3 Estimation of MA models .............................................................................................529
19.4 Estimation of ARMA models........................................................................................531
19.4.1 Non-linear LS Estimation................................................................................531
19.4.2 Maximum Likelihood Estimation....................................................................533
19.4.3 Properties of the NLS and ML estimators.......................................................536
19.4.4 Estimation of ARIMA Models ........................................................................539
19.5 Summary........................................................................................................................539
Chapter 20 Identification of Non-Parametric Input-Output Models...........................................542
20.1 Recap .............................................................................................................................542
20.2 Impulse response estimation .........................................................................................542
20.2.1 Direct Estimation using Impulse Inputs ..........................................................543
20.2.2 Estimation from Response to Arbitrary inputs................................................543
20.2.2.1 Diagonalization: Pre-Whitening the Input .....................................545
20.2.3 Regularization and Including Prior Knowledge ..............................................548
20.2.4 Estimation of IR Coefficients from Frequency Response Data.......................552
20.2.5 Indirect Estimation from Parametric Models ..................................................552
20.3 Step response estimation ...............................................................................................553
20.4 Estimation of frequency response function ..................................................................554
20.4.1 Sinusoidal Input-Based Estimation .................................................................554
Contents xv
20.4.2 ETFE................................................................................................................556
20.4.3 Estimation from Spectral Densities: Spectral Analysis (SPA) ........................558
20.4.4 Smoothed Estimates ........................................................................................560
20.4.4.1 Smoothing the ETFE......................................................................561
20.4.4.2 From Smoothed PSD Estimates .....................................................561
20.4.4.3 Welch’s Averaged Approach ..........................................................563
20.5 Estimating the disturbance spectrum.............................................................................563
20.6 Summary........................................................................................................................565
Chapter 21 Identification of Parametric Input-Output Models...................................................568
21.1 Recap .............................................................................................................................568
21.2 Prediction-error minimization (PEM) methods.............................................................569
21.3 Properties of the PEM estimator....................................................................................573
21.3.1 Consistency of PEM Estimators......................................................................575
21.4 Variance and distribution of PEM-QC estimators.........................................................581
21.5 Accuracy of parametrized FRF estimates using PEM...................................................585
21.6 Algorithms for estimating specific parametric models..................................................589
21.6.1 Estimating ARX Models .................................................................................590
21.6.1.1 AUDI: Estimating Several ARX Models Simultaneously..............591
21.6.2 Estimating ARMAX Models...........................................................................593
21.6.2.1 Pseudo-Linear Regression Method for ARMAX...........................596
21.6.3 Estimating OE Models ....................................................................................597
21.6.3.1 Stieglitz-McBride Algorithm .........................................................599
21.6.4 Estimating BJ Models......................................................................................601
21.7 Correlation methods ......................................................................................................603
21.7.1 Instrumental Variable (IV) Methods................................................................604
21.7.2 Properties of the IV Estimator.........................................................................606
21.7.3 Multistage IV (IV4) Method ...........................................................................607
21.8 Summary........................................................................................................................608
Chapter 22 Statistical and Practical Elements of Model Building..............................................611
22.1 Introduction ...................................................................................................................611
22.2 Informative Data............................................................................................................612
22.2.1 Persistent Excitation ........................................................................................613
22.3 Input design for identification .......................................................................................614
22.3.1 Pseudo-Random Binary Sequences.................................................................615
22.3.2 Preliminary Tests for Input Design..................................................................618
22.4 Data pre-processing.......................................................................................................618
22.4.1 Offsets, Drifts and Trends................................................................................619
22.4.2 Outliers and Missing Data...............................................................................621
22.4.3 Pre-Filtering.....................................................................................................633
22.4.4 Partitioning the Data........................................................................................635
22.5 Time-delay estimation ...................................................................................................635
22.5.1 Definitions .......................................................................................................635
22.5.2 Impulse Response Estimation Method ............................................................636
22.5.3 Frequency-Domain Estimation Method ..........................................................637
Get Principles of System Identification now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
