Dictionary of Occupational Titles® (DOT)

History

Selected Characteristics of Occupations

DOT Online

Social Security's Use

DOT 4th ed

References


Job Analysis

Literature

Systemic Research

Relevant Links and Articles

References


eDOT

eDOT Methodology


Job Analysis and DOT Courses

Conducting Job Analysis

Job Evaluation

Workers' Compensation

Lost Wages

Federal Employment Laws

Eliminating the Gender Pay Gap

Quantitative Methods Used in Analysis of Discrimination

Use of the DOT in the SSA Disability Determination Process

Comparing the DOT, O*NET and eDOT

Occupational Assessor® - eDOT FLSA - Professional Edition Construct Validity Resources


In August of 2001, the Disability Research Institute of Chicago (DRI) published Research Approaches for Validation of SSA's Medical Listing. SSA (the U.S. Social Security Administration) also contracted and received a preliminary report from DRI in November of 2002, "Social Security Job Demands Project, Methodology to Identify and Validate Critical Job Factors." Within these learned materials' extracts (in Maroon), PAQ Services presents notes (in Black) to illustrate how we believe the Occupational Assessor® (eDOT FLSA) meets validity challenges. We end with our comments by discussing a Daubert Challenge. Links and materials (at left) include numerous articles and sources regarding the construct of the Dictionary of Occupational Titles. The buttons below can expedite your research:

   Occupational Assessor Reference Wizard Complete Bibliography

 



Below we define various aspects of validity and related concepts, comment about the methodological issues, and present some initial ideas about the methodology that might be used to validate the enhanced Dictionary of Occupational Titles. PAQ borrows these definitions to discuss the eDOT. Simply stated, the eDOT construct is not PAQ's. It is the design of public and private researchers and has been in use for eight decades. Again, the DRI quotes are shown in maroon.)



I. Validity


 A. Face validity


  Stated as a question: Do the ... appear to relate to the ...activity? In evaluating face criteria one would consider that the ... were developed by ... experts and have been updated and have a long history of day-to-day application. (eDOT is an enhanced Dictionary of Occupational Titles, as it uses the same design as the original DOT and is intended to be used for the same purposes. The DOT has been in use since 1939.)



  B.Content validity



  Stated as a question: Do the ... thoroughly cover all relevant aspects of the conceptual domain they are intended to measure? (eDOT utilizes PAQ's Position Analysis Questionnaire (PAQ) which has been constructed to create DOT work characteristic measures. PAQ publishes the actual PAQ input and calculates and reports a (new) rate of error for each and every position each time its data is enhanced. See the separate section devoted to Content Validity, where you can review this data.)



  C. Construct validity - Selected characteristics of Occupations



  Stated as a question: Do the ... measure the underlying construct they purport to assess? Construct validity requires gradual accumulation of evidence from a variety of sources. Two subcategories of construct validity should be considered:



  1. Convergent validity: The ... should correlate highly with variables that ought to relate to the construct. (eDOT contains all 12,471 positions, plus more, with specific characteristics of occupations in use for over half a century.)

  2. Discriminant validity: The ... should not correlate with variables that ought not relate to the construct. (eDOT's "construct" is that of the original DOT. Except for the addition of "0" measures for illiteracy in 3 of the 66 work characteristic measures, no change to the DOT construct has been made). (The addition was made as jobs exist where illiteracy is acceptable.)



II.Reliability



  Reliability is an important concept that is closely related to validity. Reliability considers how consistent decisions are made by different persons and across time.

  A.Inter-Rater Reliability



  As its name implies, the agreement among different raters is used to define inter-rater reliability. A high level of agreement allows confidence that the construct being rated is clearly defined, raters understand a consistent definition and can apply that definition consistently. (Again, PAQ reports actual Field Audit input on a website where both the input data and the eDOT worker characteristics can be viewed. PAQ calculates and publishes a rate of error for each and every position on an interactive basis. Each time a measure is added, a new standard deviation is recalculated. See the separate section devoted to Content Validity.)



  B. Test-Retest Reliability



  Test-retest reliability would be revealed by the consistency in ratings over time, for the same claimant with no change in performance. (Ibid.)



  Trade-Offs between Validity, Reliability, Credibility and Administrative Efficiency The Disability Policy Panel ... described the trade-offs that are reflected in each step of the sequential process between reliability, validity, credibility and administrative efficiency. They gave thoughtful consideration to reliability and validity issues as well as to credibility (defined as the perceived legitimacy of the decision criteria from the perspective of applicants and the public) and administrative efficiency (defined as the capacity of the disability determination system to produce prompt and low-cost decisions). (This is a lengthy discussion; PAQ refers the interested reader to: "Research Approaches to Validation of SSA's Medical Listing" from which these extracts are taken.)



Daubert Challenge



  PAQ's search of the records has found no application of Daubert as it relates to the use of the eDOT (or the DOT or the O*NET) in any disability case, although this may change soon. Obviously eDOT, by using the DOT construct (and content, to start in its enhancement process), meets the spirit of any challenge (in PAQ's opinion). In one regard, the reporting of a rate of error, eDOT far surpasses either the original DOT or the O*NET. Neither report standard deviations. Neither allow researchers to review the raw data (actual job analyses). eDOT does! Two types of error rates exist:



  A. Measurement Rate of Error



  The first relates to the work characteristics, their measures and the accuracy of these selected characteristics of occupations (SCOs). PAQ's Content Validity section shows the Position Analysis Questionnaire input (or the estimated PAQ input from the original DOT measures) and calculates a mean and standard deviation for both. A rate of error is reported for the eDOT. A rate of error for the original DOT is also estimated by PAQ (from an analysis of variance). PAQ estimates the rate of error for original DOT measures because this data, if ever accumulated, has been lost from the public record.

eDOT makes the PAQ data available for public review and analysis. In contrast, O*NET does not publish its Field Analysts' input nor does it make the final O*NET measures public.



  B. Population Rate of Error



  Governments publish job family populations. (The UK and Canada utilize their Census, the US utilizes an OES survey keyed to SOC numbers, the first six of the O*NET's eight digits.) The US OES survey is of interest for Daubert because it also publishes a Relative Standard Error for population and wage numbers. The eDOT application (both Internet and Delphi PC software) is part of ERI's Job Availability project which collects data from a variety of sources, inputs this data in real time into a common Internet database, and calculates a distribution of specific jobs within an OES Job Family (for example, the percentage of Java, COBOL, Fortran and Cold-Fusion programmers within the computer programmer job family within an OES geographic area). The sources and codes used to populate the eDOT Skills Project Raw Data Content and ERI's Job Availability Survey as displayed on the Raw Data Screen (of ERI's Occupational Assessor) include the following:



 
PQ PAQ subject matter expert scored field job analyses (only past 15 years).
eD Disability Determination Analysis Cybernetic Input.
eS Administrative Law (SSA) Cybernetic Input.
eV Vocational Rehab module's Cybernetic Input.
eW Workers' Compensation eDOT input with PAQd (direct) Form.
eF US National Fair Labor Standards Act Overtime Cybernetic Input (partial analyses) with the Occupational Assessor's Professional Edition's use also contributes data to the eDOT Skills Project database, although only 11 of the 99 eDOT measures are included.
SC Internet collected skills and SCOs collected before receipt of requested salary data (CareerBuilder and other leasing job boards, SalariesReview, SalaryExpert, SalaryExpert Pro+, and SalaryExpert ePRO) all using a rotating 33 sets of 3 questions, 99 eDOT measures held in buffer until full.
SR SalariesReview - Six question access to eDOT Skills Project from ERI's patented interactive surveys; Abbott, Langer Association Survey participants, SalariesReview, ERI Salary Surveys, other surveys, PAQ US FLSA determinations.
91 Original revised DOT - 1991 values - Most of these data points rely upon work incorporated in the 1977 edition.
M7 Nine modeled analyses around the Original 1991 SCO mean value (distributed via that measure's standard error of estimate). See "Raw Data Reseeding of Original DOT Measures."
E2 New jobs added from PAQ research, ERI's Job Availability Survey, and eDOT's Subscriber Submit New Job input (eDOT users).
Other abbreviations found in ERI's Job Availability Log have included:
SV ERI Surveys - ERI researcher analyzed sources; commercial surveys reporting these specific jobs (not found in Raw Data)
US Noncopyrighted Data - OCR proxies, Form 990EOs/EZs/PFs and 8-Ks, OES, OSHA, Census, etc. (not found in Job Demands' Raw Data)
WC Workers' Compensation - data submitted by state workers' compensation agencies, including California, etc. (not found in Job Demands' Raw Data)
WZ Wizard Lookups - ERI Survey Finder - Job Availability Survey - eDOT Skills Project Raw Data - including employment and loan applicant flagging provided to lending institutions and employers (not found in Job Demands' Raw Data)
  Data is contributed into a dataset of single entry lines.

source/version, PAQ # (if any), positiontext, date, soc#, eDOT#, date,zip,oesarea,surveycompany,survey#,pop V,2000,Accountant Turf,01/24/2003,98225,530860,9999,999,1

From this data, eDOT estimates the allocation of specific jobs within a job family. For example, if 3,000 computer programmers are reported for Orange County and the percentage of Java programmers is 11%, an estimation of 333 Java programmers is made. PAQ also reports OES stated Relative Standard Error for this job family.



Whether eDOT will meet a Daubert Challenge has yet to be reported. eDOT reports a rate of error; we have found no other sources that attempt this. Indeed, the latter category, job availability, finds no evidence of any other source in existence (including the US federal government) reporting specific job populations (even estimates). Knowing the number of jobs that exist would be of value, in particular when allocating funds for training and education.


Summary



The sources at the left detail 65 years of writings and research related to the construct of the DOT. Specific potential use of these validity and reliability definitions relate to a disability assessment in the SSA's Five-Step Sequential Evaluation Process:



  1. Is the individual engaging in substantial gainful activity?



  eDOT does not apply.



  2. Does the individual have an impairment or combination of impairments that significantly limits his or her physical or mental ability to do basic work activities?



  eDOT does not apply.



  3. Does the individual's impairment(s) meet or equal a listed impairment in Appendix 1 to Subpart P of the Regulations Part 404 (Listing of Impairments)?



  eDOT does not apply.



  4. Does the individual's impairments prevent the claimant from performing his or her past relevant work?



  eDOT's work measurements may be used for comparisons.



  5. Does the individual's impairment(s) prevent him or her from performing other work?



  eDOT work measures allow the identification of other work. eDOT's job availability estimates allow a determination if this work actually exists. eDOT goes even further by listing potential employers within an industry that might employ this work/worker using publicly available records of those employers with qualified benefit and retirement plans. Finally, even example job postings are potentially available.



PAQ's eDOT Project is admittedly an ambitious undertaking. Its perfection will depend upon the data inputted by trained professional field job analysts.

*Quotes (shown in maroon) were taken from: August of 2001, the Disability Research Institute of Chicago (DRI) published, Research Approaches to Validation of SSA's Medical Listing and "Social Security Job Demands Project, Methodology to Identify and Validate Critical Job Factors."
 

Thumbnail Screenshots by Thumbshots