Realizing automation (autopilot) of construction machinery with AI

We will develop AI that automates the operation of construction machinery (crane, hydraulic excavator, etc.), industrial cranes, forklifts, etc.
We provide the creation and implementation of AI algorithms as a series of services for construction companies and construction machinery manufacturers working on automation development.

EXAMPLES OF AI-BASED AUTOMATION METHODS

By attaching cameras and sensors to navigate the environment (situation) and a module to operate levers to the actual construction equipment, AI will create and control the machine's autopilot mode.

Automation of construction machinery

Reinforcement Learning and Imitation Learning for Automation

There are two methods to create autonomous construction equipment:

The first method is "Reinforcement Learning." 
Autonomous construction equipment learns by trial and error of various maneuvers in its own environment (simulation environment or actual site).
It takes a lot of time to learn, but it can gain more control than humans.

The second is a method that uses "imitation learning."
Autonomous construction equipment learns equivalent operations based on human (skilled engineer) model data.
It is believed that it will be possible to achieve the same level of action as humans in a relatively short time.

reinforcement learning

Demonstration using simulator

Demo 1: Excavating soil with a hydraulic excavator

Utilizing the industrial simulator "Vortex Studio" (provided by CM Labs Simulations / Information Services International-Dentsu), we trained hydraulic excavators to excavate more than a certain amount of soil through reinforcement learning.

At first, it couldn't scoop any soil at all, but it gradually learned the angle of the bucket and when to move it, and was able to scoop more than a certain amount of soil.

Demo 2: Stop a crane's suspended load from shaking

A skilled human crane operator can stop the shaking of a suspended load by manipulating it when it shakes. In the same way, we have trained the crane to stop the load from shaking by reinforcement learning.

As an initial condition, the crane generates random shaking. The crane learns through repeated failures, such as moving the arm in a direction that increases the shaking, or hitting the floor with the suspended load. Eventually, the crane is able to stop the shaking within a certain period of time.

BIM/CIM LINKED CRANE SIMULATOR

DYNAMIC 3D REPRESENTATION OF CRANE HOISTING, SLEWING, AND UNDULATION TO SUPPORT CRANE STUDY WORK SUCH AS COLLISION CONFIRMATION.

This simulator enables 3D dynamic simulation of cranes on the Unity simulator using BIM/CIM data as input. This will improve the efficiency of the planning phase and facilitate the coordination of the parties involved, thereby increasing productivity. In the future, further functional enhancements are planned, such as crane lifting capacity determination and automatic planning proposals using AI.
The crane model shown in the figure is based on BIM data from Tadano Ltd. (all models except the crane model were developed by Araya)

Functions & Features

DYNAMIC MODELS CAN BE CREATED FROM CRANE BIM DATA, REGARDLESS OF MANUFACTURER.

Introduction Results

THIS SIMULATOR WAS INTRODUCED TO JFE ENGINEERING CORPORATION FOR PLANNING STUDIES OF BRIDGE SITES.

COMMENT FROM JFE ENGINEERING

THE CRANE PART INFORMATION CAN BE ASSIGNED TO ANY 3D MODEL, SUCH AS SLEWING BODY, BOOM, HOOK, ETC., GREATLY INCREASING THE NUMBER OF CRANE TYPES THAT CAN BE USED AS 3D DYNAMIC MODELS. IN ADDITION, THE ABILITY TO PARAMETRICALLY MANAGE THE CRANE MOVEMENTS OF BRIDGE GIRDERS FROM GROUND CLEARANCE TO ERECTION MAKES IT EASY TO EDIT THE CRANE MOVEMENTS AS THEY CHANGE FROM TIME TO TIME."

Video of suspended load transfer using this simulator (overhead viewpoint)

Video of suspended load transfer using this simulator (overhead viewpoint)

Video of suspended load transfer using this simulator (top viewpoint)