i3 | January 03, 2022

Smart Farming at CES 2022

John Deere’s tractor-planter can accurately place seeds and cover 500 acres in a day, or four-fifths of a square mile.

City dwellers and suburbanites usually don’t think of the farms that provide the food they eat, or the coal mining that helps generate the power to charge an electric vehicle (EV), although they may be annoyed at the road work crews slowing their commute. But for decades, all three of these — farming, mining and construction — have been at the forefront of the same autonomy and automation tech trends the auto industry is pursuing. And the innovations in those sectors are cross-pollinating with car tech. Some are being showcased at CES 2022.

AI and Sensor Fusion

Autonomy, artificial intelligence (AI) and automation are a perfect fit in the agriculture industry because “the macro factors are there” to necessitate and support it, says R. “Ray” Wang, principal analyst and founder of Constellation Research Inc., in Palo Alto, CA. “We are in the middle of a labor shortage and we have an aging and growing population. We will need to feed more people with less arable land. To do that, it’s really about precision agriculture or smart agriculture,” Wang says, adding that autonomous farming equipment factors into every stage, from planting to harvesting and tilling. Then, he says, it merges with the world of self-driving trucks for the distribution phase.

“The most exciting part is, it’s the combination of analytics, automation and AI. These three A’s are very important when you’re talking about smart agriculture.”

Wang says, by leveraging AI and cloud-connectivity in the realm of the Internet of Things (IoT) — such as smart sensors and data — the optimal time to leverage automated or autonomous farm equipment can be determined. For example, by sensing the condition of the land, it might be predicted that the best time to deploy an autonomous strawberry picking machine would be seven-and-a-half days before the crop is ripe. Thus, better use of data will yield greater productivity with the farm, even with scarce labor, he explains.

Autonomy and AI technology for farming “is certainly not new,” says Kaan Kurtural, cooperative extension specialist in viticulture and oenology (grapes and wine) at the University of California in Davis, CA (UC Davis). “It has been around for about 15 years. However, there was no economic need previously. Now, due to the increase in labor prices, there is a chronic need.”

Kurtural’s team at UC Davis works on automating dormant and in-season vineyard practices — including pruning, shoot removal, crop load management and harvesting — which account for about 56% of the labor costs on a farm, he says. “These things are what we term semi-autonomous. Someone still has to guide the tractor” through the vineyard, though the machine then automatically does its work.

“Where the AI comes in is, we can feed data (such as vineyard maps) into these machines, and they will determine management zones within the vineyards, so they can be “differentially harvested,” Kurtural says. “High quality fruit will go into one bin and mediocre quality goes into another bin, and then the harvest will be segregated.” The result can be a tremendous boost to the farmer’s bottom line because the prices of the different quality fruit can range from $300 per ton to $6,000 per ton.

The big technological leap materialized about six years ago, he adds. “Previously, we would go out, sample the vineyards, create maps, and then load them. Now we can sense on the fly, then send the signals to the machine in real-time.”

“The goal is to fuse all the data streams that are coming in,” from sensors that detect soil mineral nutrient and water content along with satellite imagery, for “a better ability to harness the variability in these fields,” Kurtural says. An average California vineyard spans 160 acres, so one area can be quite different from another, he notes.

“The next frontier is being able to document the organic carbon stored in the soil, to combat climate change,” Kurtural says. “Agricultural commodities capture a lot of carbon. Now we have ways to sense this as crops are growing, so it could be possible for farmers to bank carbon credits and sell them,” he adds.

“Food is our most intimate exposure to the environment. We transform that environment into our being and more people want to know what’s going on with their food: where its coming from and how was it processed?” says Matthew Lange, president of the International Center for Food Ontology, Operability Data and Semantics (IC-Foods), based in Davis, CA. Lange says, IC-Foods is working on the broad terms that enable technology to better understand, decipher and work with the sensor data that provides insight into food. “Sensors can provide transparency for making claims about food and abstract it to a growing method” offering proof that a crop was organically grown.

“These data sets (from farm equipment sensors) could represent in a unique combination, a concept. We can have more insights to describe what’s happening in our food, but we can’t build machine learning models over this unless we’re using the same labels to describe the various concepts that we’re talking about,” he explains. “There’s this movement to create ontologies, which are essentially computable vocabularies,” as well as a growing need for the same semantics (or meanings) to put into an ontology, “leading to deductive logic.” That could result in hypotheses about how to make farming smarter because, “the ontologies could generate code for the software that needs to do the things we’re talking about. It becomes essentially this virtuous cycle.”

John Deere farm equipment utilizes self-driving technology, GPS and robotics to dramatically speed up operations.

To this end, says Lange, IC-Foods is a partner in Icicle.AI, a National Science Foundation-funded institute operating out of Ohio State University. It’s the first seeking to “build cyber-infrastructure for AI that is itself built on top of AI. In other words, it’s AI for cyber-infrastructure for AI – AI for CI for AI. so that it gets smarter.” This involves machine learning (ML), which is itself a subset of AI, “but there will also be other methods of intelligent computing and data gathering that don’t necessarily involve ML,” he says.

“Digital agriculture” is one of three “use cases” that Icicle.AI is pursuing — encompassing, for example, self-driving tractors and smart irrigation systems informed by the weather, all enabled through connectivity, he notes.

“We’re about democratizing AI,” he asserts. “What would it look like if you could access AI for things you wanted to figure out, at the flick of a switch, in the same way you can get running water or electricity? The everyday consumer is a bit of a reach in this early stage, but food systems — and being able to rapidly pivot in the case of a disaster — makes a lot of sense and is good for the population as a whole.”

Icicle.AI, launched in October with $20 million in funding, to keep it going for five years. Besides IC-Foods, its partners include nine universities and four supercomputer facilities, Lange says.

Sensors are a window into nature that can help predict the future, and with edge computing and AI built into farming equipment, it’s possible to create a more resilient food system and devise better farming practices, he explains.

“Thinking about the challenges of agriculture — from high levels of variability, narrow windows of time, availability of labor whether skilled or unskilled — over the course of the next decade, we will have an autonomous production system” and “not just autonomy of tractors,” says Dan Leibfried, director of automation and autonomy at Deere & Co.’s John Deere Intelligent Solutions Group, based in Urbandale, IA.

Previously farmland was managed as one big block of land, Leibfried recalls. “The realities of agriculture, though, is it’s highly variable” across a field, with diverse soils and rolling topography. In the 1990s, precision agriculture technology began to manage zones of a field, then to manage at the row level of a crop. “Going forward into the future it’s going to get to a point where it’s managing at the plant level,” he declares.

Deere’s own technological progression mirrors this. In the 1990s the company began utilizing GPS to precisely document soil variability, then in the 2000s integrated it as an input to a control system for precision guidance and auto-steering that can stay within an inch of a target line on a repeatable basis. From 2011, cellular-based cloud connectivity was added to Deere’s large agricultural equipment, to “turn data into insights, using connectivity as well as the scalability of cloud-based computing,” Leibfried says. “That was the second big change.”

Now, computer vision and machine learning are being used “to even more precisely manage an agricultural field at the plant level,” such as with camera-based technology put on a sprayer to differentiate between a weed, a crop and soil, so that herbicide is applied only to the weed — reducing herbicide usage by 60% to 90% across a field, and thereby improving environmental stewardship and lowering production costs. “Computer vision and machine learning, is one of the next big technology breakthroughs for agriculture to help farmers manage at the plant level and do it at scale.”

The company debuted a tremendous, multi-ton robotic harvester (known as a combine) at CES 2019, which integrated cameras with AI to sense the quality of the crop coming through it and make automatic adjustments to the machine based on what’s detected, in near real-time. However, harvesting “is the last step” in farming that “gives the farmer a scorecard for the year” regarding crop quality and potential. In 2020, the company featured a large self-propelled sprayer outfitted with the camera-based weed sensing technology. And for CES 2021, the company used virtual reality (VR) to show a tractor with electric-drive robot seed planters spanning the width of the machine, capable of precisely laying down more than 30,000 seeds at the bottom of a trench of soil while traveling over 10 miles per hour.

At CES 2022, Deere will demonstrate at an outdoors exhibit “the next step in our journey,” with a tractor that has utility across multiple farming applications, Leibfried says. “We will have a broad focus around further levels of automation, but a strong emphasis on where we’re going in autonomy this year.”

Autonomy and automation are tied together, says Parag Garg, chief digital officer at CNH Industrial NV, the second-largest maker of agricultural equipment in the world, after Deere. Its brands include Case International Harvester and New Holland. The state-of-the-art today is “Level 4.5” — understood in agriculture to be a bit beyond supervised autonomy — the industry “will continue to be on the journey to full autonomy,” Garg says. The future of farming will comprise four megatrends: defining and enabling an ecosystem that ties different kinds of machines together with “open APIs” (application programming interfaces) that make farming digital and seamless; a digital customer experience that lets farmers buy, maintain and trade-in their machines online; automation and autonomy; and a new model for service and support required by this new digital world.

To be sure, startups are innovating in agricultural machinery, too. Take, for instance, Advanced Farm Technologies Inc., based in Davis, CA. The three-year-old company is devising a self-driving hybrid EV with an on-board gasoline-powered generator (akin to the discontinued Chevy Volt car) specifically for harvesting strawberries. It uses a stereo camera and AI to determine which strawberries are ripe and picks only those, explains Kyle Cobb, co-founder. Advanced Farm has partnered with Yamaha and Kubota, both of which are investors in the company. The partnerships are expected to help the company move toward mass production of the machine, Cobb says. Working prototypes have harvested millions of berries since early 2020, with about a dozen prototypes in fields right now, he says.

Auto industry suppliers are involved in smart agriculture, as well. In 2016, for example, Robert Bosch LLC — a Tier 1 supplier to automakers — launched a Commercial Vehicles and Off-Road (CVO) unit devoted to agriculture, construction and material handling. “We wanted to try to bring the scale of the automotive components that Bosch has as a technology provider, and bring that into the off-highway space,” says Jason Keith, CVO’s director of sales, based in Farmington Hills, MI. “Historically that was a very challenging thing, we found,” because the driving environment is very different off-road, where there are no lane markers or framed borders. This necessitated adapting automotive multi-camera systems, radar and ultrasonic sensors for these differences, he explains.

But Bosch is not offering complete automated steering systems to manufacturers like Deere or CNH that have already developed those themselves, Keith says. Rather, Bosch is concentrating on providing those sensors and connectivity technologies to facilitate other intelligent features for off-road users.

One example is NEVONEX, a connected ecosystem that enables suppliers, like seed or fertilizer companies, to provide digital services to ensure their products are applied at the proper time for the best possible outcome. To wit, the farmer can scan a barcode on a package of seeds, and the machine via a cloud interface can then be set up automatically through the NEVONEX system to spread it, and once done, upload the planting details to a farm management system. NEVONEX will begin rolling out commercially next year, Keith says. It’s in a test phase now with 30 partners including equipment makers, suppliers, dealers and connectivity device providers.

CVO will be exhibiting NEVONEX at CES in the larger Bosch booth.

While the business unit’s major focus is agriculture, it is also focused to a lesser degree on construction, and mining — utilizing the same multi-camera, radar and ultrasonic sensors, Keith notes.

The Volvo LX03 is the first real-world example of a self-learning wheel loader with the brains to make decisions, perform tasks and interact with humans.

The Toy that Became the Machine

Children love to play with toy replicas of construction equipment, such as wheel loaders and dump trucks. Now, engineers at Volvo Group have brought such a toy to full-size life, in partnership with another famous Swedish company: LEGO Group.

Unveiled in October, the Concept LX03 is a fully autonomous, battery-electric, self-learning wheel loader prototype by Volvo Construction Equipment (Volvo CE), based on a LEGO Technic model.

Endowed with “the brains to make decisions, perform tasks and interact with humans,“ it heralds advances in smart construction technology as well as toy transformation. It represents both the company’s further foray into machine intelligence and its determination to decarbonize the construction industry, and marks the first time a LEGO Technic model has been turned into a real machine.

“Together with the LEGO Group we are pushing the boundaries of both technology and imagination and the result is beyond anything the world has seen before,” says Melker Jernberg, president of Volvo CE.

Niels Henrik Horsted, LEGO Group’s head of product for LEGO Technic says, “Our two companies have enjoyed an extremely successful partnership over the years in which we have collaborated on several LEGO Technic models. But this is the first time we are making a real machine based on a model and not the other way around — and that is what makes this a truly unique project.”

Volvo CE and LEGO began exploring the future of construction together with a collaboration between engineers and designers, then expanded their project to involve a team of children. That resulted in the toy that gave rise to the machine, formally called the 42081 LEGO Technic Concept Wheel Loader ZEUX.

The Volvo CE LX03 is a five-ton wheel loader developed as a modular concept, able to be made larger or smaller with a few changes to the manufacturing process. Its computer intelligence allows it to adapt to different scenarios, make decisions in real-time and collaborate with humans. It also can complete heavy, repetitive or dangerous tasks, thus making a job site safer. It shares its driveline with Volvo CE’s L25 Electric wheel loader, so it’s also a zero-emission, low-noise machine that can operate for up to eight hours, Volvo CE says.

Caterpillar’s Cat 793F autonomous truck improves productivity, lowers operating costs, and increases safety on mine sites.

Electrification and Autonomy in Mining

“There is a broad shoreline of opportunities to leverage the technology from automotive into agriculture and construction, and likely vice versa,” says John Deere’s Leibfried. Construction is another industry vertical that John Deere serves, “and I believe that in certain applications it’s going to require multiple sensors, like radar, LiDAR and cameras. Some you’re going to be able to do effectively with cameras, and some with just LiDAR. Having capabilities across multiple technologies allows us to accelerate our journey to solve for the variety of use cases that are out there.”

Deere is a leader in both construction for earth moving and road building technologies, Leibfried says. “We’re now starting to apply the broader technology stack that [the company] has to our construction space,” to accelerate the “transition to automation and autonomy. We’ve been doing it partially over the course of the last decade,” he adds.

Autonomy is most advanced in the mining industry, asserts Karl Weiss, CTO at Caterpillar Inc., based in Deerfield, IL. In fact, Weiss says, Caterpillar has been working to automate and make autonomous mining machines for over 30 years, and has the largest installed base of self-driving mining haul trucks worldwide at nearly 500. And collectively, they’ve moved themselves over 135 million kilometers, which is 2.5 times farther than any automaker’s self-driving fleet has traveled, Weiss says.

It was Caterpillar that developed the first LiDAR used in experimental self-driving cars — albeit done in partnership with a Tier 1 automotive industry supplier. Like automakers, Caterpillar is now pushing toward EVs. “In the next five years we’ll be bringing together the electrified mining fleet and autonomy,” and “autonomy will have a significant role in how that plays out,” Weiss says.

In November, Caterpillar announced a strategic alliance with Newmont Corp., the world’s largest gold mining company, to deploy 26 first-of-a-kind battery electric, fully-connected and autonomous haul trucks by 2027 — for use in both underground and open pit operation at mines across two continents.

“It’s an exciting time because so many things are possible now technology-wise,” Weiss declares.

Subscribe to i3

I3, the flagship magazine from the Consumer Technology Association (CTA)®, focuses on innovation in technology, policy and business as well as the entrepreneurs, industry leaders and startups that grow the consumer technology industry. Subscriptions to i3 are available free to qualified participants in the consumer electronics industry.