To ensure the safety, security, and sustainability of the Netherlands’ ocean assets, accelerating advancements in maritime and offshore sensor technologies is critical. The time to advance digital adoption within traditional or digitally engaged maritime and naval enterprises – and position the country’s Blue Economy as digitally mature – is now.

In SWZ|Maritime’s September offshore special, Eric Engelbrecht, Senior Business Developer at TNO, discusses developments in digitalisation in relation to the offshore industry. He describes the path forward to a connected North Sea.

Consider for a moment current estimates of the scale of growth in the global offshore wind industry. Current projections anticipate a global increase of 228 GW within the coming decade. This equals a tenfold increase in demand throughout the entire offshore industry value chain, of which a significant amount is composed of expertise that resides in the Netherlands. Is the Dutch maritime and offshore industry poised to capture the speed and scale of this growth? Is the necessary workforce of skilled and qualified personnel available? From the viewpoint of traditional constructs for manned operations, the answer is simply no.

The Dutch Blue Economy will be forced to target its investments in nearby regions, due to an inability to operate efficiently at a truly global scale. However, with the right strategy and focus, a digital transformation agenda for the Netherlands’ maritime industry can be realised, allowing for maritime and offshore energy companies to reach millions of customers in every region of the world.

Open innovation ecosystems We are seeing in real-time that plans for enterprises to advance towards digital maturity get compressed into months instead of years for many industries. For competitive maritime and offshore companies, the demand is no different. The speed and scale of such a transition within the Netherlands can only be made possible with focused investment into open innovation ecosystems, bringing to fulfillment a Dutch industrial base poised to provide the key enabling technologies and methodologies needed to digitise its Blue Economy.

Boskalis support vessel conducting ROV operations in the IJmuiden sea locks.

It also requires shifting an enterprise’s view towards the optimisation of the digital experience for its workers, customers, and suppliers – connected products and services, which are simple, standardised, and cloud-based.

Key enabling technologies

The case for the connected worker is preeminent within the maritime and offshore space. Connected workers are operators, field workers, engineers, and managers that need real-time information, supportive working systems, and environmental (air, surface, and subsea) integration. These resources allow them to make appropriate value-based decisions in remotely secure and safe locations. Key enabling technologies for such an interface are sensors, extended reality, robotics, and autonomous platforms at an industrial scale, specified to integrate and interoperate with customers, providers, and government via predictive twins.

One example is the routine, accurate, and comprehensive monitoring and inspection required by an increasing amount of water assets. This is needed to ensure timely maintenance work and carefully monitor the condition of deteriorating assets. These activities take place both inland (for example quay walls and locks) and offshore (such as offshore foundations and subsea infrastructure). Additionally, ecological assets must be baselined, monitored, and supported to the benefit of the environment, assuring compliance and sustainability.

Key enabling technologies are sensors, extended reality, robotics, and autonomous platforms

For asset owners and governments, making a structural reliability assessment of their assets results in high maintenance costs and potentially insufficient resources. These costs drive the need for a more effective standardised way to perform underwater monitoring and inspection, optimising operations throughout the value chain.

Offshore subsea operations: challenge and opportunity

Meanwhile, offshore operations involve increasingly hazardous environments such as the deep sea, resulting in escalating costs and complexity for the health and safety of field workers including technicians and divers. To ensure cost-effectiveness in the subsea environment, to increase capacity, and to provide a safer alternative to diver-based operations, offshore asset owners and operators increasingly make use of Remotely Operated Vehicles (ROVs) and Autonomous Underwater Vehicles (AUVs), which can be equipped with a wide variety of imaging sensors. Acquiring high-quality data with all available sensors currently demands extensive expert knowledge and labour intensive, manual postprocessing.

Even if a complete set of high-quality data is available, data interpretation is challenging, because of the different sensor types used for underwater inspections. While cameras are useful for close-up inspection, their range is often less than a half metre. Because of this, acoustic sensors are used for large area inspection and navigation at the cost of lower grade resolution.

To build a complete picture underwater, often both optical and acoustic sensors are needed

Data collected underwater can also be subject to large geo-referencing errors, limiting the added value of using both acoustic and optical sensors. This is particularly so when different platforms are used to capture the sonar and camera data.

To build a complete picture underwater, often a combination of optical and acoustic sensors is needed. Currently, these imaging modalities are not readily available in a fused fashion, which complicates unmanned operations and asset inspection. Technical challenges like these demand solutions that provide a frictionless experience to customers on a global scale. These solutions need to take into account the job that needs to be done and the necessary connected products or services have to be created rapidly. TNO is keenly focused on developing such key enabling technologies and methodologies – partnered with industry and government – to demonstrate the clear Return On Investment (ROI) and a path towards commercialisation for the entire value chain.

Subsea robotics and SURE

Imagine a 3D underwater world based on a virtual environment that is enriched in real-time with multisensory input. It is a new and intuitive interface for anyone active in underwater applications. With the right investment, it can be available to industry and government within the coming years, perhaps sooner. Currently within TNO, the SUbsea enhanced REality (SURE) research programme, part of the Open Innovation Center “i-Botics”, aims to make underwater inspections and operations more efficient and effective from remote facilities at sea and ashore.

i-Botics, founded by TNO and the University of Twente, is an innovation hub for interaction robotics. It develops knowledge and technology for value-added robotic solutions based on requirements from stakeholders. Industry partners in the full value chain as well as government are crucial for the development and successful implementation of these solutions.

Sonar, camera, LIDAR, and other sensors are being utilised as input to provide an underwater blended reality environment with haptic robotic interfaces. This fusion of camera and sonar data is not straightforward. It requires data of a different physical nature to be merged and projected in one shared representation. Data collected underwater can be poorly geo referenced, needing to be warped to ensure correct dimensionality and overlap, while guaranteeing that the relevant intrinsic properties of the data are not lost.

Furthermore, the amount of data collected using high-resolution (high-frequency) sonars is tremendous, requiring advanced data processing (edge computing technologies for data filtration and high power computers) to allow the data to be visualised in 3D, geometrically correct, and enabled for full operator interface.

Artificial Intelligence

Artificial Intelligence (AI) is applied to enhance the underwater blended reality environment and provide additional information for:

  • object recognition and classification;
  • assistive visual cues for manoeuvring, manipulation, and other operations;
  • automated anomaly and defect detection, supporting both infrastructure and ecology; and
  • automated change detection, supporting both infrastructure and ecology.

This underwater blended reality environment is being deployed as real-time support for ROV/AUV operators, inspectors, and asset owners in industry and government.

TNO, together with leading Dutch small, medium, and large enterprises, and governmental asset owners, is developing more harmonised, higher quality, cost-effective inspection and monitoring operations underwater. Experts in the fields of underwater data acquisition, data processing, 3D visualisation, data interpretation, human-machine interaction, and automation are working together to advance subsea operations with all the relevant stakeholders.

SURE will develop knowledge and technologies for the full value chain in both offshore and in-land underwater applications. Advantages for respective value chain members include:

  • technology providers – increased knowledge and potential technological solutions in sensory fusion, data processing robotics, and enhance human-machine interfaces for blended reality;
  • ROV/AUV and system integrators – capacity to develop novel (tailored) control and inspection interfaces for customers, with tailored architectures/frameworks;
  • service providers – improved operational efficiencies, safety, and operator experiences leading to reduced costs and optimised asset management services; and
  • end-users – reduced operational downtime and enhanced capacity.

TNO, Boskalis, and Rijkswaterstaat have initiated the first phase of SURE to achieve safer and more effective damage monitoring and inspection of underwater assets. Results from this phase will be published in subsequent articles.

IJmuiden locks test site

As part of the first phase of SURE, the data needed to build the 3D model has been acquired by Boskalis and Rijkswaterstaat at the IJmuiden lock test site. TNO provided input on how they carry out the experiments in such a way that it enabled the fusion of the sonar, camera, and LIDAR data. Additionally, TNO executed the data processing, fusion and creation of a blended reality digital twin of the IJmuiden harbour.

Dr Frank ter Haar, researcher at TNO, demonstrates the sensory fusion and immersive reality of SURE for operations at the IJmuiden sea locks.

‘SURE represents the realisation of a needed technical capability. State-of-the-art underwater technology is tested in a pilot environment that is representative for operational conditions,’ says Sander Steenbrink, General Manager Corporate Research & Development at Boskalis. ‘We aim to use the combined data to make better, informed decisions about inspections, repair and maintenance operations. The expertise at TNO on data fusion has proven to be instrumental in reaching the project objectives.’

3D virtual world

Rijkswaterstaat has provided available data before the start of experiments and is acquiring additional data with their surveying equipment. This data is being fused in a 3D model to demonstrate the added value for asset inspection and enhanced operator support. The developed 3D environment is being examined in the context of asset integrity monitoring and inspection. Feasibility for automated asset monitoring and inspection, including change detection and damage classification applications, are being assessed with consideration for the large variety of damage that can occur to underwater structures.

SURE Sensory Fusion Display, IJmuiden sea locks.

Joris Vijverberg, Manager Infrastructure and Mobility at the State Innovation Programme of Rijkswaterstaat confirmed: ‘It is important that underwater work is carried out safely. That’s expensive. In addition, there are many possibilities to improve the quality of the information produced by the inspections. This allows asset management to be carried out cheaper and more sustainably. As soon as the techniques developed in SURE are sufficiently market-ready, we want the companies that carry out inspections on behalf of us to use these techniques.’

Algorithms were developed and applied to make the various data modalities geometrically correct

Because asset inspection underwater requires both local high-resolution details and large coverage overview, acoustic sensors (large coverage, but coarse resolution) and cameras (high resolution, but limited visibility) are being utilised for this purpose. The images with this article show examples of 3D reconstructions based on acoustic sensors and/or stereo cameras developed to-date. The camera and sonar data were fused into one 3D virtual world. Algorithms were developed and applied to make the various data modalities geometrically correct, which is essential for enabling data-fusion.

Rapid development needed

Advances such as these are what will allow the realisation of the connected worker and digital maturation of Dutch offshore and maritime enterprises. This offers clear benefits to industry, in both the near and long term:

  • reduces human error;
  • increases productivity;
  • more time-efficient;
  • cost-effective (reduced operational spend);
  • improves the safety of workers;
  • enhances decision making;
  • allows 24/7 monitoring of assets;
  • prevents harmful incidents;
  • faster learning and training for qualification and advancement; and
  • ecological monitoring and environmental compliance.

Such benefits help enterprises enhance their operational models, allowing for greater speed and scale to sustainably and responsibly capture market share. While key enabling technologies and methodologies within sensory fusion and robotics have been illustrated, transformational progress toward predictive twins will rely on rapid development in these additional areas:

  • platforms – Artificial Intelligence and Machine Learning via autonomous systems (air, surface, subsea);
  • interfaces – 3D digital twins and sense-enabled (Haptic) human-machine interoperability;
  • devices – embedded IoT enabled smart devices and sensor miniaturisation; • cloud and edge computing – embedded high power computing (HPC);
  • ocean information and communications technology (ICT) – 5G and dedicated low space orbital satellite networks; and
  • cyber/information security – blockchain (secure exchange of data and certification/compliance).

Together with partners and through its open innovation centres, TNO seeks to advance the Dutch maritime and offshore innovation agenda in each of these areas. Its goal is to assist both industry and society in the implementation of a digital transformation agenda for the Netherlands’ Blue Economy.

Picture (top): Boskalis ROV at the IJmuiden sea locks outfitted with Sensor Suite.

This article was originally published in SWZ|Maritime’s September 2020 issue and written by Eric Engelbrecht.