What Is a Time of Flight Sensor (ToF)?

The time of flight sensor (ToF) has a peculiar name. It doesn’t necessarily mean it will calculate the time a flying object is in the air, nor does it measure the precise time an object takes off from the ground. Before understanding what a ToF sensor does, it’s essential to understand what ToF actually is. ToF measures the time it takes for a physical object to travel a given distance through a medium. Typically, this measurement can determine velocity and path length, but it can also be used to learn about an object’s dimensions.

A Time of Flight sensor can use all the information gained using ToF principles for applications such as robot movement, human-machine interfaces, – like the second-generation Kinect sensor for the Xbox One – smartphone cameras, machine vision, and even Earth topography. While all these uses aren’t exactly the same, the information provided can serve all their purposes. Now that we’ve established what a ToF sensor can do, it’s equally important to determine what they consist of, how they generate the information, and, finally what specific purposes this information can serve for in the world of robotics.

How a ToF sensor works with light reflections. Image via Wikipedia

What Makes Up a ToF Sensor?

From a general perspective, a ToF sensor isn’t a device that requires decades of research to understand. It’ll consist of a few parts but none of them are particularly obscure or hard to piece together.

The first part is the lens, which, given that it’s essentially a camera, is pretty easy to understand. The lens itself, like any other camera, gathers the reflected light since it cannot produce any light by itself nor can it acquire depth signal from ambient light. According to a scientific study by Subhash Chandra Sadhu for Texas Instruments, ToF cameras have “have special requirements to be met with while selecting or designing the lenses.” While the rest of the study goes on to explain the specifics (which he explains very well), it’s important to understand these limitations if you want to fabricate your own ToF sensors in the future.

An example of a ToF sensor. Image via Quality Magazine

Also included in the ToF camera package is the integrated light source that keeps the seen, well, lit. Considering that all the light must come from the sensor, it’s equally important to make sure that no outside sources of light – like sunlight – disrupt the image intake.

Then there’s the image sensor, the centerpiece of the ToF camera. The sensor does the heavy lifting, storing all capture image information, including the time it takes for the light to travel from the integrated light source to the object and then back.

Finally, there’s the interface, which shows the data captured. It’s the less showy aspect of the ToF, but, hey, it’s still essential!

How Does a Time of Flight Sensor Work?

The Time of Flight sensor is able to capture depth information for every pixel in the image captured. It is mainly used for machine vision applications and advantages include the sensor’s compact construction, its relative ease-of-use, a precise accuracy of approximately 1cm, and high frame rates.

There are 2 principle ways in which a ToF sensor can determine distance and depth.

The first is is a ToF sensor based on light pusled sources. This form will measure the time it takes for a light pulse to travel to from the emitter to the scene and then back. Once everything has been measured and taken, through the magic of mathematics and algorithms, the distance and depth of all the objects captured by the sensor are calculated and determined.

At Seeed Studio, they mocked up a graphic that simply yet accurately depicts how the process works.

Easy enough, right?

This graphic explains more clearly how the light refracts from the object back to the sensor and how that measures the distance from each relevant point. Image via Seeed Studio.

The second way is a ToF based on continuous waves which detects the phase shift of reflected light. The modulating amplitude creates a light source inm a sinusoidal form with a known frequency.The detector then determines the phase shift (a shift when the graph of the sine and cosine functions shift left or right from their standard position) of reflected light.

Once this process happens, more math happens as well, determining the distance and depth of all the objects captured by the sensor.

 While the end results of both methods are similar, their journeys differ. The illumination of the entire scene, regardless of the method, will make it possible to determine the distance and depth of each object scanned by the sensor – all in a single shot.
 
The result? A range map in which the pixels encode the distance to each point on the captured scene.
 
Over at Melexis, they showed a depth image of a man in a car. The colors are represented as follows:
– The blue sections indicate that the point(s) is(are) far away
– A red section indicates closer proximity to the sensor
Image via Melexis.

The Advantages and Limitations of the Time of Flight Sensors

Like any set of technological tools, there are upsides and downsides.

 

Some of the clear advantages of using ToF sensors for 3D measurements are the following:

  1. Higher resolution captures.
  2. Real-time capabilities – no need to wait days for a result.
  3. Works in low light conditions – even no light is possible.
  4. The costs aren’t particularly high.
 
On the other hand, its limitations are worth considering, just in case your needs aren’t lined up with what a ToF sensor can do. They are as follows:
  1. The presence of scattered light due to unwanted reflections.
  2. External bright surfaces that are close to the camera can quickly scatter too much light into the lens, creating artifacts.
  3. ToF distance measurement requires light that has been reflected just once.
  4. If a light has been reflected multiple times, it can lead to the distorting of the measurements. These multiple reflections are usually caused by corners and concave shapes.
  5. Ambient light and sunlight make it more difficult to capture outdoors (sunlight causes the saturation of sensor pixels)

In What Manufacturing Contexts Can You Use ToF Sensors?

ToF sensors are highly practical in numerous applications including logistics, factory automation, and autonomous robotics and vehicles.

For logistics, ToF sensors can help guide robotic arms for packaging assistance, box filling, stacking, volume scanning, and labelling. A pick-and-place case study conducted by Lucid at Pensur, an engineering company, looked into how their 3D vision systems allowed for a far more efficient process and freed up valuable time for employees who were stuck doing the menial job day-in-and-out.

In the context of factory automation, ToF sensors can guide robots to find and pick up objects and place them where they need to be. Think of a car assembly. Nothing changes from car to car, but the ToF sensors will point out where everything is and everywhere they need to be.

ToF sensors can also be used in the context of maritime navigation in that the sensors can use AI-based object recognition. This is done to increase security on boats during sailing by detecting objects that may conflict with the ship’s path such as fishing boats, buoys, and debris, which can’t be detected by using only the ship’s radar.

IDenTV showed a brief example on their YouTube page of how these cameras can work and how quickly they can detect objects even at far distances.

Finally, for autonomous robots, a ToF sensor can help a robot plan and execute a task all on its own. Whether those processes consist of sanding, powder coating, or batch painting, the ToF sensor can help the robot understand each object’s specific dimensions and, with the help of the right software, can execute each task necessary by knowing where to start and stop.

ToF sensors are at the core of AutonomyOS™ and AutonomyStudio™. They are key to the first step: 3D perception, helping autonomous robots figure out what they need to do in real-time. Contact us today for more information.

4 Key Takeaways From FABTECH 2022

FABTECH has once again come and gone and this year’s rendition offered, for the first time since the pandemic, a chance to see everyone’s faces again. With hundreds of companies and tens of thousands of exhibitors and attendees alike roaming around in the Georgia World Congress Center, there was a diverse set of robots, machines, and approaches to manufacturing automation. But what stood out the most?

During our one-week stay in Atlanta, we had enough time to not only talk to some fascinating manufacturers but to peruse the rest of FABTECH to see who and what caught our attention. Here are our 4 biggest takeaways from FABTECH 2022:

An Abundance of Cobot Welding

FABTECH is known for the wide variety of exhibitors showing off new machinery, robots, and software but sometimes, you’ll find several companies displaying the same thing with different bells and whistles. This year, we saw a high number of cobot welders (and welding-adjacent exhibitors too) and counted a total of 37 of them across the three exhibit halls. 

Based on the number of people who came to our booth and asked us about lot-size-one welding solutions (it’ll come!), we can definitely understand the need for new welding solutions, with an average age for welders topping 55 and little helping seeming to be on the way. After all, we listed recurring problems in welding and how to solve them, but current solutions are still mostly suited for higher volume weldments not necessarily the futureproof way of enhancing this process – although they do play their part.

There are always new easy-teach tools being released with the goal of producing consistent high-quality welds, but even with these new additions, the cycle time challenges will continue to persist. Specifically, with lower volume productions, some of the challenges such as teaching and repeatable fixturing remain constant. While they make the end result better, the road to it doesn’t get much easier. Despite this, the demand at FABTECH for autonomous welding solutions shows just how much pain still exists in that world. It is abundantly clear that people are looking for small, low-cost solutions with a willingness to “figure things out” if it does not address each and every requirement in their production process. 

Subscriptions: A Path to Being Recession-Proof?

Every manufacturer, machine builder, and integrator knows how difficult it can be to keep up with a recession. With cash becoming more of a luxury during tough economic times, how do you continue to improve productivity when keeping staff becomes increasingly strenuous? How do you justify spending on robotics solutions when most of them come at a high initial cost all while needing operators?

Instead of investing hundreds of thousands (or more) all at once on integrating a high-mix solution, subscriptions to software-based solutions will lessen the burden of spending all that cash in one spot. With a subscription, it’s clear that you won’t own the software you’re using to solve your problems. What it will do, however, is spread out your bill over monthly or yearly periods. If a subscription consists of an hourly rate far below that of skilled labor burden rates, then suddenly the cost to finish a process becomes more palatable and sustainable for manufacturers looking to save costs and maximize efficiency – and you finally get systems that can address an indefinite variation of parts, which cannot be addressed with traditional robotics today. 

Naturally, there may be exceptions, but very few. Companies that function primarily on liquid cash – a rarity for most – won’t see their cash flow optimized by subscriptions. A company with a usage-based subscription can go up and down on operating expenses and they can shift their usage based on market conditions. 

Subscriptions offer “recession-proofing” because your operational expense can fluctuate with demand, instead of capitalizing highly complex integrations where the payback may not be realized if a recession happens earlier than expected. This kind of whiplash is still what the industrial sector still suffers the most from, despite every other sector having become more agile through flexible payment models. Even if you pay more in the long run, you can not only get more value and a solution that grows to meet more needs – you can also pay less when you need to, a critical feature when urgent cost cutting is required.

Subscriptions are surefire ways of making sure your processes can continue in spite of an economic downturn. By effectively managing recurring payments and enabling more stability and consistency in your skilled labor force (read: more productive labor, less lay-offs), then you can rest easy knowing that you’ll be able to hold the fort during tough times. 

Labor Shortages For All?

The two words that came up the most during FABTECH were “labor shortages.” Of course, this isn’t surprising. Since the start of the pandemic, it’s been especially difficult to replace skilled workers performing manual labor. Now that we’re further removed from the pandemic as ever, manufacturers need the same level of productivity they had prior to 2020.

In theory, people want their cycle times to be as close to zero as possible. Granted, no action will ever be instantaneous, but the more we lessen cycle times, the more efficient everyone will be. If manufacturers add robotic or automation solutions to their setups, they need these new solutions to be efficient. If an automation solution is complex and adds more work to the operators, then what kind of time are you actually saving? 

By switching to smaller, simpler, more modular, and mobile solutions – machines that can function with infrequent or limited oversight from operators – the incremental cost of using your capital will be significantly stabilized, maximizing the speed with which you can get an outstanding payback. As well, by freeing up the operator’s time, they can move on to other tasks that require human input in a different way. Essentially, they can solve more with less.

Skilled Labor, On-Demand: The Power of AutonomyOS™

Though the above points vary in nature, they all share some sort of connection. After all, manufacturers, while they all differ in their work – all face similar problems. Labor shortages, market saturation, and money management can make or break a company. To find a solution that can tackle all these problems, it might benefit your company to take a look at AutonomyOS™, a software that can essentially act as skilled labor on-demand. 

At our booth, we saw several people ask us about processes we expect to support in the future. Our sanding solution with DIY Robotics was well understood from the jump, with manufacturers grasping how the process works quite easily. There is an abundantly clear need for autonomy as a solution to shortages of all kinds.

With AutonomyOS™, you can say goodbye to having skilled workers working around the clock to finish a list of items to be powder coated, sanded and – in the future – deburred, welded or more. Thanks to a OpEx-friendly subscription, and a wide array of ready-to-go processes, all manufacturers will need to do is press a button to have their skilled robots do the tough work (just like the sanding solution at our booth!). It is quite literally skilled labor, on-demand: use what you need and pay accordingly. With the ability to have robots and cobots alike perform tasks without the need for intricate setups by operators, you can unlock productivity at faster rates than ever before.

The advantage of our hourly model – where you only pay for your software subscription in operation – means that you can stick to minimum commitments and add usage through a token system. This overflow can help you manage the ups and downs of production utilization, but also help maximize the utilization and Overall Equipment Effectiveness (OEE) of all the other equipment you’ve capitalized in your facility. 

At the same time, we don’t make you rent equipment – you can still buy that yourself, finance or rent through a third party – but the autonomy which makes it so powerful for high-mix manufacturers is available on-demand and works flexibly to help you meet your growing needs.

The best part? The more you use it, the faster your payback on your entire robotic system.

Industrial Space Continues to Grow and Adapt

The combination of reshoring, labor shortages, and financial uncertainty are forcing everybody to get lean and efficient – fast. This doesn’t mean big massive system integrations – small becomes beautiful. This also means simplicity, minimal training and high-speed payback are essential.

Transformative technology is the only way to do this – old models simply will not do. While many were not at FABTECH and busy minding the store at home, so many more at the show were truly ready for new ideas, new modes of production, and new ways of doing business, because they feel the pain the most. We know many may have stumbled a few times or more in their journey through automation, but fortunately, autonomy is here for you today.

With an AutonomyOS™-enabled robotic machine, you can say goodbye to labor shortages and maximize your efficiency. With the ability to set up behaviors to execute tasks such as paint spraying, sanding, welding, and more, you’ll find all the flexibility you want for your manufacturing needs. Contact us to learn more

Recurring Problems When Programming Robots and How to Move Past Them

Table of Contents

With the increasing abundance of robots, you need a relative increase in engineers to set them up for each company. So how complicated of a process is programming a robot?

The truth is that it’s pretty difficult! There are many factors to consider, including the robot’s capabilities, the space surrounding the robot, how the robot will move, and of course, the programming language necessary to program the robot. 

Robots are essentially motion recording devices that allow you to generate a procedure. The procedure in question – varying from company to company – will likely only be relevant to a single function in which the part processes or other variables remain constant at all times. While this method could still be useful for a number of processes and companies, sometimes it can be done in a smoother fashion.

Let’s go over which problems are the most common and how, exactly, they affect development.

Programming Languages

Like with any verbal language, programming languages exist in abundance. Each language is used for a specific purpose, but it isn’t feasible for a programmer to learn a plethora of different languages. 

Manufacturers and the robots they develop use different languages. What ABB uses isn’t quite what FANUC uses, and what FANUC uses isn’t what Universal Robots uses. Germán Villalobos, an AI and robotics engineer explained in a LinkedIn article that each manufacturer will “have more than [three] different brands installed in their cells and production lines, which further complicates their robot programming training.

Automation engineers and programmers would essentially have to learn each robot manufacturer’s programming language if they want to work efficiently on their assigned robot. However, learning an entire language, let alone a programming language is an arduous task that will take hours to learn and even more to master.

While new programming languages emerge every few years, the main ones still reign with JavaScript dominating. Chart via Devskiller.com

High Costs, Low Time

Based on various reports, including the aforementioned LinkedIn article, it can take over 70 hours to properly learn just how to develop a simple application in any given programming language. Multiply that by the number of robots you have with their own individual languages and add the time it’ll take to complete an automation system, and suddenly you have weeks of training that needs to get done.

The cost of investing in training for every employee who needs to learn an additional language could be astronomical depending on the sheer number of employees. As well, you have to factor in equipment like cameras, computers, and well the robots themselves if you’re buying new technology.

Villalobos estimated that for each person trained, it could cost up to $15,000 per person. That number only gets higher with each new brand of robot a company acquires. To avoid spending all this money on training, it’s important to find alternatives such as hiring employees who are familiar with programming specific types of robots, or simply moving away from programming altogether and opting for a behavior-based robot instead.

While learning isn't inherently a bad thing, it can be a burden for companies trying to keep their employees up to date on new programming languages only suited to a specific robot brand.

The Complexity of Programming Robots

The myriad of programming languages and the high cost might lead you into thinking programming robots is complex. It most certainly is, but those factors are merely small factors in the complexity scale.

Robotics companies don’t even hide how complicated it can be to program a robot. In fact, DIY Robotics has a page dedicated to some recurring problems a programmer will encounter when working on robotics projects. In brief, they describe problems during the programming process that include misunderstandings of the physical limitations and capabilities of the robot. However, to ease the burden, they suggest using tools that each robot manufacturer offers to lessen the burden.

Villalobos continued in his article that robot programming is too difficult to do properly and efficiently work on robots. He argues that robot programming “has the same bases of computer science plus the difficulty of handling the different mechanics of robot arms, electronics controllers with software that differ between manufacturers; and that are also highly customizable for different processes and different industrial quality and safety standards.”

With so many variables to consider along with the rigidness that programming brings, it can be overly complicated to properly program a robot within a reasonable amount of time to perform specific tasks.

Academics and Programming Robots

The complexity of programming robots is not only known to manufacturers but academics have also noted this. In a study conducted by Eleonora Bilotta and Pietro Pantano for the University of Calabria back in 2000, they analyzed a variety of problems including the “difficulties in programming the robot control [and] the organization of the program in relation to hardware, software, behaviors, and performance design in robotics.”  Specifically, they focus on robotics in relation to teaching control to children. While this isn’t exactly manufacturing, their discoveries and criticisms of programming are pretty similar to those encountered in that field.

Across the study, Bilotta and Pantano argue that the current method of programming robots could be better and lean toward modern proceedings including bottom-up robotics and behavior-based robotics. And though 22 years have passed since the study’s publication, some of their criticisms still remain relevant.

They describe some of the pains they encountered with programming, including the lengthiness of the process as well as every external factor that could come into play when trying to execute a specific action. Instead, they prefer to try and work through behaviors.

“From the programming point of view, the behavior space of the robot is defined by the locations the robot can reach (or by the set of actions it has to exhibit in the physical space) and by the transition between those locations. Even if the robot can attain a nearly infinite number of states, it is better to design a useful behavior space in which the programmer limits him/herself to a small number of states,” they state. 

They imply freeing up time thanks to behaviors better understanding the capabilities of robots while not taking up as much time to get them running. Even more than 20 years ago, behavior-based robotics was seen as the future. 

Behaviors for All!

Robot programming is an intricate skill, craft, or trade – call it what you want – but it needs to evolve. In software programming, new languages emerge every decade, or even every few years, either rendering older languages obsolete or confusing older engineers by adding to the amount of knowledge they need to amass to do their job.

Open-source solutions include Swift, Rust, and Kubernetes which only gained popularity over the last few years. They’re far from being the most dominant, but their emergence isn’t negligible.

Machine builders and integrators are not programmers by trade – they’re designers. Designers need simple solutions. They need to be able to do more with fewer (or even) lines of code.

Behavior-based robotics is on the way to becoming the best way for robotics to move forward without the crutch of having to adapt to new languages every time they come out. Instead of relying on preset calculations that can handle a fixed process, behavior-based robotics adapts to its environment to perform a series of heterogeneous tasks. 

They can adapt using sensors essentially telling the robot what the piece is, its dimensions, and how it can best perform the task it was set up to do. All this is done through an interface that is more user-friendly and that takes away the need to parse through hundreds of lines of code.

Setting up an autonomous robot for the first time seems like the dawn of a new era, but it can also be misleading. To the untrained ear, the word “autonomous” sounds like it can do anything based on the power of AI alone, or something along the lines of Wall-E from the Disney movie. While performing any task might be a tall order right now, an autonomous robot can perform a specific task given it’s been programmed to do so. The real question remains: do you still want to be programming robots well into the future?

An example of how behavior-based robotics could work for an open-source project on Github.

With AutonomyOS™ and AutonomyStudio™, your flexible automation cell will be as powerful as ever. With the ability to set up behaviors to execute tasks such as paint spraying, sanding, welding, and more, you’ll find all the flexibility you want for your manufacturing needs. Contact us to learn more

Choosing The Right Equipment For A Flexible Automation Cell

Automation can provide relief to manufacturers looking to subvert the ongoing labor shortage, but it can also be confusing when it comes to the initial implementation of an autonomous robotic system. One of the first things you need to know when setting one up is if it’ll be a fixed or flexible automation system. Once you’ve established that, it’ll then be the time to figure out what kind of equipment you’d need to better serve your factory. 

With so many different suppliers, machines, and setups, you want to make sure that whatever decision you make, it’s a comprehensive one – one that ticks all the boxes for your manufacturing plant and one that can secure a more productive and efficient future. 

Fixed Vs. Flexible Automation

When you decide to automate, the work in which you automate will fall into one of two types of automation: fixed or flexible. For example, if your business is focused on the assembly of the same pieces over and over again at a high volume, you’d be more inclined to try a fixed automation system. However, if your factory is High-Mix, fixed automation won’t exactly work.

What is “High-Mix” Manufacturing? It is generally defined as any manufacturer or production that processes more than 100 different SKUs in batches of fewer than 1000 each year – basically, a lot more variation than mass manufacturing.

ABB Robotics unveiled their latest look into flexible automation cells at Automatica 2022. Image via ABB.

If the pieces you manufacture fit the High-Mix description, then the type of automation you’re going to need is called flexible automation. In short, fixed automation will serve a single purpose at a high rate, and flexible automation will help serve multiple pieces at a lower volume.

Preparing yourself for a fixed automation system is more straightforward than a flexible automation system even if it’s more limited. Preparing yourself for flexible automation is a little bit more complicated, but with the right research and understanding, the extra effort will be far more fruitful. 

Let’s say you have a High-Mix production and you’re looking to install the perfect flexible automation cell, what exactly do you need to ensure that your cell is as comprehensive and complete as possible?

The Right Equipment for a Flexible Automation Cell

Flexible automation systems aren’t always cut-and-paste. Some serve different purposes. While some automation cells will fit the traditional need of having a robot perform certain tasks until completion, other cells will simply be a stackable storage system that will help organize the inventory to help get end products to customers faster. Considering the differences in systems, not every piece of equipment listed below will be useful to every specific flexible automation system. Depending on what type of cell you’re looking into, the following pieces may help bring a greater understanding as to what you may need or want when setting up your flexible automation cell.

 

The Robot

A robot arm, while not vital, can help accelerate the production process thanks to mature AI software.

Technically, robots are not always central to flexible automation. However, they can greatly help because they are more articulated and versatile tools than other pieces of equipment for flexible automation such as large rail inventory systems. Due to their restrictive nature, it’s easier to turn to an autonomous robot with powerful AI alongside it. Sometimes, this will require the robot to be programmed with the help of a capable robotics software like ROS. Other times, you may want to remove programming altogether and get behavior-powered software that will allow the robot to learn about the parts it will work on and execute each task for each individual piece properly and efficiently.

These processes don’t even have to be all the same. You can alternate between painting, sandblasting, deburring, and more if you need to. There exists a myriad of options that will help you get a robot. Companies like FANUC, Universal Robots, and Yaskawa all have a deep catalog of robots that can meet your needs.

Omnirobotic’s AutonomyOS™ is the world’s only platform for truly autonomous manufacturing. Using 3D Perception with AI-based Task Planning and Motion Planning, manufacturing engineers and integrators can configure autonomous robotic systems for value-added processes that reduce labor shortages, increase productivity, save energy, waste and rework and allow manufacturers to achieve more consistency and flexibility in production than ever before.

The Cameras

Cameras, sensors, and localization are not necessarily essential tools for flexible automation, however they provide one benefit that makes installing them worthwhile. By installing any of these, you will eliminate the need for programming jigging.

With a proper set of cameras and/or sensors, the robots will be able to properly perceive any piece that passes through them. Once those pieces have been reconstructed digitally, an autonomous robot can then understand how to perform an action. If you’re setting up cameras and sensors, however then that likely means you need objects to pass through them to the robot.

The Material Handling

If the pieces you need to work on need to pass through cameras, then your flexible automation cell will need a conveyer of some sort. While there are a myriad of conveyor options to choose from, you’d need to determine which, based on the space you have, properly fit your factory floor. The conveyor isn’t the most glorious or most coveted piece of equipment but it’s a necessary one to facilitate the process.

The Space

Okay, space isn’t quite a piece of equipment, but having a large area to work with is certainly helpful to the cause. Depending on the system you have, the space you would need would vary. If you only have space for a small conveyer belt, a couple of cameras, and a robot, your flexible automation system could work, but be limited in what it can execute in a day.

Sprimag, a company focusing on automated coating systems, detailed what their coating cell would look like. Based on their mockup, you can that it’s a long one. They have a large amount of space, but it allows the system to go through several different stations. The robot, more or less placed in the center, has enough room to move around without risking a collision with any of its surrounding walls.

The cell’s loop-like structure will reduce the floor space necessary for material handling. Its versatility in regards to size is the flexible automation cell’s biggest advantage here. With an easy-to-place design, it won’t restrict the other essential parts of the factory.

Sprimag's flexible automation cell isn't overwhelmingly big and it allows for a streamlined process thanks to multiple stations.

Another Flexible Automation Cell Example

Manufacturing company Liebherr detailed a rotary loading system that allows a robot to pick and place objects in a circular cell. In a detailed account of what this system entails, Liebherr states that “the individual workpieces lie in these bins in chaotic order. The core of the Liebherr bin picking system is an intelligent piece of software that compares data from 3D visualization of the bin contents with the actual CAD data of the workpieces being searched for and detects the correct parts.”

For a system like this, you would need more than just the robot, cameras, and conveyer. You would also need an intricate storage system that will work in conjunction with the aforementioned pieces of equipment. What might seem like a disorderly mess is actually a fully functioning system for the robot and for the flexible automation cell itself.

Why Is Flexible Automation So Important and Popular Right Now?

The uptick in flexible automation can be associated with several points. For one, there’s been a trend toward mass customization in manufacturing. As more manufacturers deal with High-Mix environments, their pieces aren’t always homogeneous and benefit from the flexibility that automation can provide. High-Mix manufacturers need flexibility to use automation properly for their needs.

As automation evolves and matures, the industry is expected to leave a smaller environmental footprint. With sustainability becoming a larger focal point for manufacturers, it’s important to realize just how much automation can benefit both manufacturers and the environment.

As well, with space being such a scarce commodity in manufacturing, it’s important to make the best use out of whatever space is available in any given factory. As zoning restrictions tighten up, saving space becomes the best and most efficient way to not have to change locations. With the right flexible automation cell, using the least amount of space to achieve the most amount of work is the simplest solution. Sprimag and Liebherr have managed to do it, so maximizing space is certainly within reach.

Valin Corporation showed the overall cost effectiveness of a flexible automation system versus a fixed automation system.

Different Equipment for Different Needs

Not every flexible automation system is universal, naturally. Each cell will be tailored to each company’s needs, therefore, leading to a myriad of different equipment configurations. With companies like Sprimag and Liebherr detailing what their ideal flexible automation cells will look like, their needs aren’t their peers’ needs. It’s vital to assess the type of automation and choose the right equipment to go with it.

With AutonomyOS™ and AutonomyStudio™, your flexible automation cell will be as powerful as ever. With the ability to set up behaviors to execute tasks such as paint spraying, sanding, welding, and more, you’ll find all the flexibility you want for your manufacturing needs. Contact us to learn more

What Is An HMI And Why Should You Care?

Many new pieces of technology always seem to have long names with abbreviations to simplify their pronunciation. Sometimes, they’re easy to decipher like the widely-known AI – or artificial intelligence. Other times, they refer to technologies with specific uses and would only be known to the users who handle them daily. The Human-Machine Interface (HMI) is one of those pieces of technology.

An HMI is a user interface that acts as the communicator between a user and the machine, computer program, or system with which they are interacting. It’s a broad term, sure, and can be linked to several home appliances (something like the ill-fated Wii U that Nintendo sold in the 2010s), but we typically refer to HMIs in an industrial context for larger machinery.

A Brief History of the HMI

While the modern HMI has been around since the 1980s, its origins can be traced all the way back to 1945 when the Batch Interface allowed perforated punch cards to be inserted into the machine to calculate the number of hours employees worked. It was a primitive interface that was non-interactive but it laid the groundwork for future interfaces that would adapt to future technology.

 
A card with punched holes enters into the batch interface to transfer data from a census into statistics. Image via Wikipedia.

In the following decades, as technology vastly improved, graphical user interfaces began to sprout in order for machines to perform the jobs they were built to do. The Command-Line interface allowed users to take a bit more control. They could enter commands in the prompts to perform certain tasks. While they first appeared in the 1960s, their uses grew in the 80s when Windows Disk Operating System (DOS) became a staple in user interfaces. 

Obviously, technology has grown up quite a bit since then. As interfaces became more user-friendly, HMIs grew with them. HMIs are a natural extension of the Graphical User Interface (GUI) and allow total control of machinery in industrial contexts. They are primarily used in several manufacturing processes.

So What Does An HMI Consist Of?

An HMI is essentially an advanced user interface to help manufacturers control their machines efficiently to execute a task. In the interface, HMIs can display data, track production times, color code messages, and, of course, start and stop the machinery at play. If it sounds like an advanced remote control, well, that’s because it is, sort of.

These days, HMIs can function like tablets in the sense that there’s software with a touch-screen allowing you to communicate with the machinery however the programming allows. They aren’t always limited to the tablet form, however, as they can also simply be applications on traditional computers.

An HMI provides you with an in-hand hub to view all the goings-on in your factory.

Who Are The Primary Users Of An HMI?

 

To answer in a single sentence: manufacturers. Manufacturing processes can differ from factory to factory but a common trait between them is their use of machinery. More specifically, those who will see the most use out of HMIs are engineers, systems operators, and system integrators.

These workers can use HMIs to see data in real-time, change the speeds of different machines, or simply monitor the machines remotely. HMIs can save the time of its operators by giving them a hub in hand that can allow them to monitor the machinery across the factory. By removing the tedious walking back-and-forth across the factory, operators can use their time more efficiently.

HMIs are becoming more commonplace in manufacturing workplaces. As automation grows in popularity, so have its companion screens,

What Types Of HMIs Exist?

 

There are three different types of HMIs that you can use.

The Push Button Panel

This one is straightforward. Instead of having an assortment of buttons across a machine for different actions, this HMI will round them up in one digital panel so that they’re easily accessible. It makes the lives of the operators easier by streamlining the number of buttons you need to push (effectively zero if you set everything up properly).

The Data Handler

 

As you may have guessed, this HMI will handle, well, the data. These types of HMIs will offer feedback about a machine’s performance using the data it collects after performing tasks. Be sure to have a screen large enough to see all the information the HMI will throw at you because it will come in the form of graphs, charts, and other forms of visual representation of the data it collected.

The Overseer

 

This type of HMI isn’t as menacing as its name would let on. The Overseer requires a Windows computer to operate. Essentially, this HMI monitors and controls entire sets of machines across a factory. As its name suggests, it oversees the entire operation rather than one set machine. Consider it the big boss of HMIs if there were a hierarchy.

Where You Can Find HMIs

 

HMIs can be tricky to find, but they’re not inaccessible either. You can find them on automation-focused websites like Automation Direct or WiAutomation. Since these companies are focused on automation, it’ll be easier to find a brand that will suit any given manufacturer’s needs.

Third-party resellers also exist on eBay and AlieExpress, but you should exercise caution if you’re looking for HMIs on these sites. Naturally, they won’t necessarily offer the same quality or customer experience that a dedicated company will provide.

Do You Need An HMI?

 

HMIs are essential controllers for manufacturers who want to perfect and streamline their production. By centralizing everything through the interface, operators can shift their resources toward more important and vital tasks, rather than wasting time turning machines off and on, verifying their output, and overseeing the entire operation by manually checking each station.  With that in mind, if you don’t already have HMIs set up in your factory, it may very well be time to get your hands on some to maximize your company’s efficiency.

 

With AutonomyOS™ and AutonomyStudio™, using an HMI is key to ensuring your automation processes function the way you expect them to. Monitor all your equipment such the autonomous robot, the 3D perception cameras, and the software itself from the tip of your fingers using an HMI. Contact us to learn more

ROS: How Well Does it Address Manufacturers’ Needs?

The first time you see a robot perform a specific action, it can be quite awe-inspiring. Seeing robots like the Personal Robot 2 (PR2) clean tables and fetch drinks is certainly a sign that the future is now. Though the concept of having a robot understand what it needs to do is fascinating, how does it actually know what to do and how to do it?

There isn’t a universal answer to this. Robots have, for the longest time, been able to simplify some elements of programming thanks to robotics middleware such as Urbi, OpenRDK, and ROS. Though these platforms all offer different advantages and limitations, ROS stands out from the crowd thanks to one thing: its open-source nature. ROS’s repository is free to access, meaning that anyone who’s interested in programming robots can start with this middleware for free.

How ROS Came To Life

The Robot Operating System, more commonly known as ROS, started as a project at Stanford University by Keenan Wyrobek and Eric Berger. During the time in grad school, the duo had noticed their peers were wasting way too much time trying to program robots – Wyrobek even heard people say they had spent four years trying to make a robot work with no success – and decided to create a universal, open-source platform that would allow developers to share their knowledge.

“People who are good at one part of the robotics stack are usually crippled by another[…]” said Berger in an interview for IEEE Spectrum. “Your task planning is good, but you don’t know anything about vision; your hardware is decent, but you don’t know anything about software. So we set out to make something that didn’t suck, in all of those different dimensions. Something that was a decent place to build on top of.”

 
Since 2018, robot installation numbers have fluctuated namely due to the pandemic causing significant changes to the labor market. Graph via IFR.

In a separate guest editorial by Wyrobek for IEEE Spectrum, he specified that he had seen developers spend 90% of their time re-writing other people’s codes, with the other 10% allocated to innovating. Afterward, Wyrobek found donors to help fund the building of 10 robots and shipped them off to 10 different universities in order to have teams of software engineers build developer tools that would allow other developers to innovate and build on the software. Essentially, Wyrobek was tired of seeing developers attempt to reinvent the wheel each time, so he and Berger wanted to simplify everyone’s lives.

How You Can and Can’t Use ROS

On its own, ROS can’t really do much. There are vast libraries of packages included in the ROS repositories, but ROS itself only provides the canvas on which developers can program and execute their desired tasks.

Using ROS, developers can build the three main components of a robot: the actuators, sensors, and control systems. These components are then unified with ROS tools, namely topics and messages. The messages are used to plan the robot’s movement and, using a digital twin, developers can ensure that their code works without having to actually test it on a real robot.

These messages can travel throughout ROS using nodes, which is essentially an executable file within a ROS package. Each node is registered to the ROS Master, which sets up node-to-node communication. All this technical information to say that programming is an essential part of ROS. Developers and programmers have to code each action they want the robot to perform. Without ROS, this would be a daunting task, since developers always tend to reinvent the wheel. With ROS, however, this is a much simpler task thanks to its open-source nature.

ROS allows developers to simplify the job by using nodes to register requests to the robot and how exactly it will respond to them.

ROS succeeds in providing a canvas for its developers due to its large community size. While other robotics middleware, like URBI, aim to solve the same problems, there was one key difference in their success. URBI was an expensive software to license, and while developers still used it, it failed to build a community similar to ROS’. With a large community comes more tools for developers to share. Consequently, more projects could be pushed to completion in record time. 

In fact, the robotics middleware has become so widespread that, as per Bloomberg’s reporting in 2019, 55% of robots shipped by 2024, over 915,000 units will “will have at least one ROS package installed, creating a large installed base of ROS-enabled robots.”

Additionally, Lian Jye Sue, Principal Analyst of ABI Research claimed that “the success of ROS is due to its wide range of interoperability and compatibility with other open-source projects.” The more ROS expands through community-based packages, the more adoption rates for ROS will climb in the future.

ROS’ free entry point allowed developers from anywhere in the world to start tinkering with different projects and upload them to ROS’ repositories whenever they feel comfortable with its status or if another developer wants to take a chance and try to improve upon it.

A look at ROS' user interface running on a UBUNTU system (Image via ROSIndustrial)

The Limitations of ROS

When things are free, they tend to have some serious trade-offs. For a project with the breadth and depth of ROS, it’s understandable that it has its limitations. Developers aren’t paid when they upload their packages on ROS’ repositories, nor are they compensated for keeping them updated. Updates to the ROS platform are done regularly, but they rarely, if ever, increase the range of tasks it can accomplish. As stated earlier, open-source middleware like ROS is built to help, not reinvent the wheel.

While ROS can do a lot, its limitations can severely affect a company trying to think outside the box or simply trying to narrow down the effectiveness of its product. One of the main downsides of ROS is the potential lack of updates for certain packages. If a certain company has been working on a package but the project for which the package was made is nearing its conclusion, then updates afterward will become scarce or non-existent. The packages are left to die and can become obsolete quickly. If other developers are using these packages, then their product might suffer if bugs arise with no one to patch them.

Another area in which ROS suffers is its lack of compatibility with computer operating systems – it only works on Ubuntu. (Its successor, however, works on Windows and Mac as well but ROS 2 is far from a finished product and doesn’t offer the same consistency as ROS.) Ubuntu is not a hard real-time operating system, which means ROS could become obsolete quickly depending on industrial robotics needs. As the middleware uses more power and space, there’s no guarantee of real-time control.

Finally, ROS lacks support for micro-controllers and embedded chips – it has to run on a computer. The only real alternative for this is to run ROS on Raspberry Pi (and similar type) boards.

Though the number of flaws and limitations of ROS isn’t necessarily high, they are impactful. Still, if a company has a more narrow and focused idea of what needs to be done, then they should be mindful of these caveats. A platform like ROS was never meant to please everybody, but for a company with simple goals or for a student trying to acclimate themselves to the world of robotics, ROS can provide a solution.

Who Uses ROS?

According to ROS’ website, hundreds of companies, from startups to Fortune 500 enterprises, have downloaded over 500,000 different ROS packages for use on their projects. One Canadian company, in particular, succeeded in using ROS to develop their robots. Clearpath Robotics, founded in 2009, develops several robots based on ROS and are programmable using ROS right out of the box.

One of their most popular ROS-powered robots is the Jackal, an unmanned ground vehicle that can autonomously drive itself around a multitude of different terrains. It’s an entry-level robot, but one of the most widely used ROS-powered vehicles at the moment. With over a decade of success and usability with ROS, Clearpath Robotics is even making the switch to ROS’s successor, ROS 2, which aims to fix all of ROS’ limitations. 

 
Clearpath Robotics uses ROS and ROS2 to ensure that their deployed robots continue to develop and execute complex processes.

But it’s not just Clearpath Robotics using ROS, companies like Fetch Robotics and TurtleBot use the middleware to fill their different needs. Where the former focuses on developing robots designed for warehousing, the latter develops inexpensive, personal robot kits made more for enthusiasts and researchers, rather than whole solutions for a given industry. 

The versatility of ROS can benefit a myriad of different companies in different industries, but it’s not quite the world-changing plug-and-play solution it aims to be.

No matter how a robotic system is configured, most often an HMI will be required to make it easy for operators to manage - ROS doesn't necessarily make that process easy, however.

Enter the World of AutonomyOS

In contrast to the open-source middleware that is ROS, there exist a plethora of proprietary platforms designed for more specific uses. Omnirobotic’s AutonomyOS™ is a middleware meant to simplify and widen how robots are being used. While they both aim to achieve similar results, AutonomyOS™ flips the script by removing the need to code – something that still drives ROS.

By removing the lengthy coding process, AutonomyOS™ allows better resource allocation. Gone are the days of spending countless hours trying to find the perfect code to make the robots execute the desired tasks. The logical question to ask after reading this is “How does it work if no one is required to program it?”

Before the robot executes its actions, it needs to know what object it will be working on first. In order to analyze the object, it must first pass through a set of 3D perception cameras that will digitally reconstruct it and make it visible with AutonomyStudio™, the integrated development environment that allows for the configuration of a system in a virtual space. Though 3D perception can be costly, Omnirobotic enables integrators to deploy 3D cameras using HDR-enhanced sensor fusion, effectively eliminating the need to adjust camera parameters.

With AutonomyOS™, setting up the behaviors of the robot is essential to executing a task. That means no more wasted time on programming movements.

Once the reconstruction is complete, that’s when AutonomyOS™ shines the most. AutonomyOS™ includes a built-in task planner that can interpret any process model and can plan the desired motion to execute the tasks at hand. Using HTN planning, scenario exploration, and behavioral patterns that the end-user can design themselves, AutonomyOS™ can convert the specifics of the object’s part positions and overall geometry into usable toolpaths.

When the toolpaths are ready, AutonomyOS™ can then generate a proper motion for the robot to execute the necessary actions. Several elements are considered when planning out the proper motion, such as managing collidable spaces, avoiding singularities and joint pressure, and streaming motion through a robot controller for real-time production workflows.

Where ROS Can't Compete

AutonomyOS™ can be primarily used by High-Mix manufacturers for a variety of different applications like paint spray processes, welding, and sanding. What is “High-Mix” Manufacturing? It is generally defined as any manufacturer or production that processes more than 100 different SKUs in batches fewer than 1000 each year – basically, a lot more variation than mass manufacturing.

For AutonomyOS™ to analyze and understand the task it needs to execute, it just goes through the steps listed in the previous section and, well, does what it needs to. ROS, on the other hand, would have to be programmed to understand the shapes and technicalities of each piece it needs to work on.

With a good set of behaviors, AutonomyOS™ can execute a large number of functions

Let’s say a factory needs to paint over a batch of items – say stools, desks, and drawers. Let’s also presume that there are at least 5 different models of each item. If you use ROS to get a robot to paint over them, then you’d be required to program the robot to understand the shapes and sizes of each item, as well as to go after odd forms and intricate spaces to maximize the surface area onto which the robot is painting.

AutonomyOS™, though, will execute these tasks after having analyzed the items with its 3D perception cameras. Then, using AutonomyStudio™, the end-user can set up the appropriate behaviors to ensure that programs will be properly executed – and this before the robot has even begun moving.

All Good Things Have A Cost

ROS has its fair share of uses. Without repeating what was listed above, it’s clear that, up until a certain point ROS can help develop automated systems. Who it helps is more important than how it helps however. Its limitations are succinctly explained above, but it can be especially useful for a company with limited resources and funding to get their feet wet with automation. Given its free entry point, ROS is more of a learning software than a software that can solve a plethora of manufacturing problems.

AutonomyOS™ doesn’t share the low-cost entry point but its uses far exceed that of ROS. As well, unlike ROS packages, AutonomyOS™ won’t become obsolete because a developer has stopped working on their project. AutonomyOS™ has a monthly subscription fee but with that comes a platform that continues to grow, enabling support for more machines and robotic systems far into the future.

ROS vs. AutonomyOS™: A Uneven Battle

AutonomyOS™ expands the scope of what a robotics software can do for manufacturers. That doesn’t mean ROS is bad, it just means that, as free middleware, it limits itself given that there are no developers paid to create new packages. It’s a community-driven project that, even with constant updates, can’t revolutionize the robotics industry. AutonomyOS™ is more advanced in nature, but is also for those who are ready for full robot automation in their factories.

With AutonomyOS™ and AutonomyStudio™, it’s never been easier to deploy an autonomous robotic system. Using 3D Perception with AI-based Task Planning and Motion Planning, manufacturing engineers and integrators can configure autonomous robotic systems for value-added processes that allow manufacturers to achieve more consistency and flexibility in production than ever before. Contact us to learn more!