Part 3: Methodology

Part 3: Methodology Jessica Abramovic Tue, 03/09/2021 - 10:57

The NSC developed the Australian Skills Classification using a mix of machine learning and human judgment and drew on different data sources, including O*NET and existing skills classification systems in Australia, for the development of the Classification. The department’s employer surveys, Australian job advertisement data, and education and training course documentation were used for the validation and refinement purposes.

3.1 O*NET data and the adaptation to Australian Skill Classification

3.1 O*NET data and the adaptation to Australian Skill Classification Jessica Abramovic Tue, 03/09/2021 - 10:57

The NSC considered different existing classifications with the following key principles to develop the skill classification:

  • could be adapted for the Australian context
  • is data driven
  • would identify skills that are transferable across occupations
  • is comprehensive
  • identifies trending skills
  • is dynamic.

Among those existing classifications, the American classification O*NET stood out with these principles.

O*NET is a rich database containing information on all American occupations through annual surveys of American workers since 2000.

To adapt O*NET data to the Australian occupation classification, we first developed a mapping system between O*NET occupations (the Standard Occupational Classifications (SOC) used in the United States) and the ANZSCO occupations (the Australian and New Zealand standard classification of occupations).

By borrowing from O*NET and mapping to ANZSCO using the concordance table, we then developed a structured skill classification, describing job requirements for the ANZSCO occupations.

3.2 The structure of the classification and its coverage

3.2 The structure of the classification and its coverage Jessica Abramovic Tue, 03/09/2021 - 10:58

The NSC identified three essential categories of skills:

  • core competencies: non-specialist skills commonly used in all occupations (sometimes called ‘soft skills’ or ‘employability skills’) - skills such as communication and teamwork which underpin successful participation in work
  • specialist tasks: work activities a person undertakes specific to a job
  • technology tools: a technology, such as software or hardware, used within an occupation.

The skill classification covers 600 occupations which are a mix of ANZSCO 4-digit and ANZSCO 6-digit occupations. The decision to roll ANZSCO 6-digit occupations up to ANZSCO 4-digit was made on a case-by-case basis. Considerations included:

  • multiple ANZSCO 6-digit in ANZSCO 4-digit code map to the same O*NET occupation
  • there being no substantial differences between the 6-digit occupations in their tasks, education and licencing, or industry distribution
  • very few job listings for a given ANZSCO 6-digit code over a 5-year period (using Burning Glass Technologies job advertisement data).

3.3 Core competencies

3.3 Core competencies Jessica Abramovic Tue, 03/09/2021 - 10:58

Core competencies are the basic building blocks common across most occupations and industries. They describe a set of non-specialist skills gained in early life and schooling and provide a base to further develop skills and specialties. Popular terms for these include ‘foundation skills’, ‘common skills’, ‘soft skills’, ‘core skills’ and ‘employability skills’.

Using desktop research and consultation with relevant stakeholders and subject matter experts, the NSC identified 10 core competencies required for every occupation in Australia (see Table 1).

The NSC aligned the core competencies to the definitions of foundation skills typically used in the Australian vocational education system. Specifically, the Employability Skills Framework developed by the Australian Skills Quality Authority (ASQA). The minor differences between the Commission’s 10 core competencies and ASQA’s foundation skills were recommended by education system experts.

Score for core competencies 

The 10 core competencies are required in every occupation across the whole labour market, but different occupations require different levels of core competencies. To derive the competency required for a skill within an occupation the Commission mapped the 10 core competencies to values from three different O*NET categories of data that offered the most relevant proxy for each of the ten core competencies:

  • Skills
  • Work Styles
  • Work Activities

The O*NET score values range between 1 and 7, where the higher the value, the higher the level of the skill is needed to perform a job. The other 7 anchor on the 10-point scales were derived from the Australian classifications and ratings systems and modified to fit the context. We used Australian references, such as the Australian Qualification Framework and Australian Core Skills Framework to fill out another 7 examples across the 10-point scale. An example of anchor values for one core competency is shown in Table 1.

Table 1: Ten point scale for the core competency “writing”
Core competency Description Competency Description Value Value Description
Writing
 
O*NET Skills Communicating effectively in writing in a way that is appropriate for the audience 1 Write name and address on a membership form, copying another document
2 Write everyday workplace specific vocabulary and abbreviations, e.g. product names
3 Take a telephone message
4 Write a job history as part of a job application
5 Prepare a standard operating procedures document
6 Write a memo to staff outlining new directives
7 Write a detailed literature review
8 Write a legally binding contract for services provided by one business to another
9 Write a novel for publication
10 Write a thesis on metaphor, syntax and grammar in nineteenth century novels

 

3.4 Specialist tasks

3.4 Specialist tasks Jessica Abramovic Tue, 03/09/2021 - 10:59

The specialist tasks are designed to describe day-to-day work within an occupation. While these skills can be transferrable across occupations and sectors, unlike core competencies, they are not universal.

We used O*NET’s Detailed Work Activities (DWA) as our starting point for technical tasks. There are approximately 2,000 Detailed Work Activities in the O*NET taxonomy. Each of these represent a distinct work activity performed in a job.

To bring the DWA into the Australian context, we manually removed or adapted tasks not commonly performed in Australia – for example we took out snow ploughing. We then modified the task to translate its academic descriptions to language that Australians are more likely to use daily. The modified tasks were called the specialist tasks thereafter.

It is important to note that several specialist tasks described the same or a very similar skill in different words. To facilitate the transferability among occupations, we clustered the specialist tasks into higher level groups using a mix of machine learning and human checking methods. The underlying intention is that if an individual can perform one task from a cluster, they can likely perform the tasks in the same cluster as well.

We used 3 different clustering algorithms, k-means, Affinity Propagation and Spectral Clustering algorithms followed by a customised cluster refining process. We then used a similar process to further group the specialist task clusters into families, to give us a true taxonomy of skills, rather than an occupation driven taxonomy which includes skills. We ended up with 29 task families, 279 task clusters and 1,925 specialist tasks.

Table 2. Example of top 10 specialist tasks for Cooks, by time spent
ANZSCO Code ANZSCO Title Specialist tasks % of time spent on task Specialist cluster Specialist family
221111 Cooks Cook foods 26% Undertake food preparation Food services
221111 Cooks Check quality of foods or supplies 9% Monitor food or nutrition quality Food services
221111 Cooks Assess equipment functioning 7% Inspect, test or maintain equipment or systems Quality control and inspections
221111 Cooks Cut foods 7% Undertake food preparation Food services
221111 Cooks Prepare foods for cooking or serving 5% Undertake food preparation Food services
221111 Cooks Coordinate activities of food service staff 5% Coordinate food service activities Food services
221111 Cooks Clean food preparation areas, facilities or equipment 4% Clean work areas or dispose of waste Cleaning and maintenance
221111 Cooks Inspect facilities, equipment or supplies to ensure conformance to standards 4% Inspect products, equipment or facilities Quality control and inspections
221111 Cooks Arrange food for serving 4% Undertake food preparation Food services

Working hours for specialist tasks: Task utilisation score

To analyse how specialist tasks are utilised by an occupation, we calculated working hours allocated to each task during a day. This also helped to rank the popularity of tasks according to working hours allocated to them.

While O*NET specifies each of the DWAs associated with an occupation, it does not provide a metric of the relative time spent in each activity for each occupation. In deriving the time spent for a DWA, we applied a similar method to that detailed in AlphaBeta’s ‘The Automation Advantage’ (2017).

We used the task ratings survey carried out by O*NET that asks respondents how often they perform each task in their role, ranging from ‘yearly, or less’, to ‘hourly, or more’. O*NET reports for an occupation the share of respondents and their implied frequency for a task. We derived a task utilisation score, or the working hour allocated to a task, for each occupation and task pair. This score is computed by taking the average of implied yearly frequency for a task weighted by respondent share.

Table 3: Task implied yearly frequency
O*NET Frequency Implied yearly frequency Rationale
Yearly of less 1 Once a year (one time)
More than yearly 2 More than once a year (2 times)
More than monthly 12 Once a month (2 times)
More than weekly 48 Four times a month (48 times) (12 x 4)
Daily 240 At least daily, 5 times a week (240 times) (5 x 48 working weeks x 1)
Several times daily 480 At least twice daily, 5 times a week (480 times) (5 x 48 working weeks x 2)
Hourly or more 1,920 At least 8 times a day, 5 days a week (1,920 times) (5 x 48 working weeks x 8)

3.5 Technology tools

3.5 Technology tools Jessica Abramovic Tue, 03/09/2021 - 11:00

Technology tools are a technology, such as software or hardware, used within an occupation. Common technology tools, such as search engines and email, are featured across most occupations, and these are captured in the core competency of digital engagement. The remaining technology tools are highly specialised and occupation-specific, such as computer-aided design and carbon monoxide analysing equipment.

The data for the technology tools is originally sourced from a combination of ‘software’ and ‘tools’ found in the O*NET occupation data for the United States Standard Occupational Classification.

These occupations have been mapped to the Australian occupational classification (ANZSCO), and the technology tools have been adjusted for the Australian context by cross-reference with the Burning Glass Technologies Australian job ads data.

While O*NET contains a vast dictionary of technology tools, there are three significant drawbacks:

  • it was not developed for the Australian context
  • the list is extensive with over 16,000 tools (which is contrary to the idea of transferability)
  • it does not provide a level of importance for the skill.

We only wanted to consider tools that are in common use in Australia, so we used Burning Glass job advertisement data to filter out low relevance skills. For example, for Accountants we removed ‘Sage 50 Accounting’ (software popular in the USA) because it is minimally referenced in Australian job advertisements. However, MYOB is in high demand in Australia despite being absent from the O*net taxonomy, so we added it to the Classification.

This process required significant data cleaning and natural language processing. We used fuzzy matching for skills which are in both taxonomies. We then manually searched for digital skills in Burning Glass which are not in O*NET, such as MYOB, Xero, etc. Then we conducted an extensive review of the results to ensure the matchings and the name of the skills were correct.

As a parallel refinement process, we also used other data sources to identify whether O*NET captures all the technology tools. This involved examining results from the NSC’s ‘Survey of Employers Recruitment Experiences’ which asked employers the digital skills they use in their work, as well as exploring popular skills in Burning Glass Technologies data. The research identified several skills for supplementation.  

The technology tools were then manually checked to confirm that they met the definitions of technology tools we had determined. Some were found to not match this definition and so were manually removed.

Lastly, individual technologies with similar functionalities were aggregated together to produce the final technology tools by using the technology family-level of O*Net. For example, the technology tool ‘Accounting software’ consists of Xero, MYOB BusinessEssentials, Fund accounting software, Tax software and more. Noting the challenges of the O*NET and Burning Glass matching, aggregation makes the results more robust as well as improving transferability between occupations.

Ranking the technology tools

The technology tools are ranked according to an intensity score which measures their prevalence in job ads using Burning Glass. For a technology tool we measure the proportion of job ads in an occupation that require that technology tool. For individual technology tool examples within a technology tool, we measure its prevalence across all occupations. 

This approach allows us to rank the prevalence of technology tools within an occupation, and of technology tool examples within a technology tool. In some cases, however, technology tools were manually supplemented and not available in Burning Glass data and have an intensity score of zero, leading them to have the lowest possible rank.

An example of technology tools used by a specific occupation is shown (see Table 4) as well as examples of technology tools commonly used in Australia (see Table 5).

Table 4. Example of Technology Tools by Occupation
ANZSCO Code ANZSCO Title Technology Tool Technology Tool Ranking
221111 Accountant (General) Enterprise resource planning ERP software 1
221111 Accountant (General) Accounting software 2
221111 Accountant (General) Data base reporting software 3
221111 Accountant (General) Data base user interface and query software 4
221111 Accountant (General) Financial analysis software 5
Table 5. Technology Tools Examples
Technology Tool Technology Tool Examples Technology Tool Ranking
Accounting software MYOB BusinessEssentials 1
Accounting software Fund accounting software 2
Accounting software Intuit QuickBooks 3
Accounting software Xero 4
Accounting software Tax software 5

Deduplicating technology tools

Since O*Net maintains such a large taxonomy of individual technologies, in some cases their groupings can have significant overlap, resulting in technology tools that are highly related or even duplicate. This presents a problem for consistency between occupations – for example, the O*Net occupation ‘Cashiers’ has the technology ‘Point of sale POS software’, while the occupation ‘Travel Agents’ has the technology ‘Point of sale POS terminal’, ultimately limiting their transferability. In the Skills Classification we’ve attempted to reconcile these inconsistencies by combining technology tools that are deemed too similar.

Table 6. Examples of Duplicate Technology Tools
EFTPOS and card reading machines GPS receivers
Electronic funds transfer point of sale equipment Global positioning system GPS receiver
Bar coding software Vehicular GPS
Point of sale POS terminal Route navigation software
Point of sale POS software GPS receivers
Point of sale POS receipt printers Mobile location-based services software
Magnetic stripe readers and encoders  

Common technology tools

Several technology tools are so universal in 2021 that they are likely to be used by most or all occupations.

Rather than being individually listed against each occupation, we felt these common technology tools were best represented through the Digital Engagement core competency. This has the effect of ensuring the remaining technology tools only refer to more specialised technologies that are likely to be meaningfully different between occupations. It also prevents exaggerated transferability between occupations when they share only common technology tools.

The common Technology Tools included in the Digital Engagement core competency are:

  • Email and calendar software
  • Word processing software
  • Spreadsheet software
  • Presentation software
  • Search engine and information retrieval software

3.6 Validation exercises

3.6 Validation exercises Jessica Abramovic Tue, 03/09/2021 - 11:02

As part of the development of the Australian Skills Classification, staff from the NSC (and the Department prior to the creation of the NSC) undertook various validation exercises to ensure the accuracy Classification prior to a public release.

Independent market testing

As part of the validation activities, Nous Group was engaged to conduct independent market testing of the Australian Skills Classification. The survey investigated if the skills associated with a sample of 22 occupations in the Skills Classification match employers’ understanding of the skills required.

22 occupations were chosen for testing as this project aimed to provide initial insights into the Skills Classification rather than validate it in its entirety. Test occupations were agreed with DESE and were selected on the basis of occupation size. This was to encourage a larger survey response, and to ensure coverage of multiple industries.

Responses were concentrated on two occupations, which has given us good insights about the skills classification for these occupations. The concentrated responses likely reflect the level of engagement from their skills organisations. Small numbers of responses were also received for 11 other occupations.

Broadly, the results showed that there was alignment between employers and our Australian Skills Classification on the core competencies and specialist tasks. There was a misalignment on Technology Tools, which prompted us to revisit our methodology to improve the data.

Core Competencies improvements

The core competencies element of the Skills Classification was reviewed with the aim of improving relevance to the Australian context. 

Escalier McLean Consulting were engaged to support our work improving the linkages between our core competencies and the Australian Core Skills Framework (ACSF) and the Australian Qualification Framework (AQF). Based on the outcomes of this exercise, the core competency anchor values were improved.

Technical validation

Quantium were engaged to validate the overall technical approach and determine if the Australian Skills Classification we developed is reflective of the Australian labour market.

Overall, Quantium found that the Australian Skills Classification has met all the requirements of the technical assessment framework, however, several refinements to the methods and outputs have been identified. As part of the continuous improvement of the Australian Skills Classification, the NSC is considering these refinements.