What is machine vision? Everything you need to know

Machine vision is a technology that enables automatic inspection and analysis for applications including automatic inspection, process control, and robotic guidance by using image processing. It’s important to know that when mentioning machine vision it can be in reference to many different technologies, software and hardware products, integrated systems, actions, methods, and expertise.

Machine vision is a technical capability that is integrated with existing technologies in new ways and applies it with the aim to solve real-world problems.

Machine vision is a systems engineering discipline and can be considered distinct from computer vision which is a form of computer science and not done through a tangible piece of hardware such as a vision box or camera attached to a robot.

Machine vision is the body of a system and computer vision is the intelligence of the system, similar to how a computer is a frame for what goes inside such as the computer chips that power up the computer.

Without computer vision, machine vision can’t work as it’s the brains behind processing the information. It’s important to note that when computer vision technology advances it increases the possibility of potential applications for machine vision increases respectively. ClearView Imaging makes the valid point that computer vision can process images that might not be photos or videos and instead be an image from a thermal or infrared sensor, a motion detector, or other sources.

Machine vision systems have been in operation since the 1950s and it was between 1980 to 1990 where the technology really started to take off and grow in popularity.

Machine vision is becoming increasingly popular within industrial automation environments while also becoming more frequently used in other industries such as security, autonomous vehicles, food production, packaging, and logistics while also being included in robots and drones.

Machine vision can be integrated with technologies such as deep learning and machine learning to help businesses using the technology to understand data better and optimise the business for higher efficiency with an example being how BMW uses the technology alongside AI and machine learning to increase efficiency.

Definitions of machine vision

Machine vision has amassed various definitions for what it is and how it works. It’s important to note that machine vision is in no way associated with image processing which is a process where the output is another image. The information that is captured by machine vision will be translated into a form of data to make sense of it such as the identity, position, and orientation of the object(s) that are being captured by the machine vision system.

An example of a broader definition is one given by Automated Imaging Association (AIA) where they explain that machine vision encompasses all industrial and non-industrial application in which a combination of hardware and software provide operational guidance to devices in the execution of their functions based on the capture and processing of images.

SearchEnterpriseAI gives a narrow definition of machine vision by saying it’s “the ability of a computer to see; it employs one or more video cameras, analogue-to-digital conversion (ADC) and digital signal processing (DSP). The resulting data goes to a computer or robot controller. Machine vision is similar in complexity to voice recognition.”

Imaging inspection and sorting

The primary uses for the technology are imaging-based automatic inspection, sorting, and robotic guidance. The technology can be applied to a robot for it to spot where to put things or where to pick things up also known as pick and place while it can also be applied to part of an end-to-end solution at multiple points of the system to spot, inspect, and detect the required parts throughout the system.

The technology can be used with a spectrograph camera which adds color to the images and for when the objects are going through inspection and sorting. However, this method does come with delays as it takes longer for the machine to process the data due to higher detailed images.

Indeed, machine vision systems can be designed and implemented into a system in a bespoke manner to match the application requirements. The requirements for the system are likely going to be different if the application is for the food and beverage industry compared to if it needs to be used for the automotive industry.

A machine vision system can inspect and sort a wide range of objects and items across a broad range of industries including automotive, electronics and semiconductors, food and beverage, road and vehicle traffic or intelligent transport systems (ITS), medical imaging, packaging, labelling, and printing, pharmaceuticals, scientific development, and TV broadcasting.

Methods and sequence of operation

This is the first step in the automatic inspection sequence as the camera needs to see the object or item that it will be capturing and sending on to the processor to process the image. Depending on the level of detail required will change what camera is needed for the image capture which can range from a standard camera and lense to a 3D camera.

Machine vision software packages and programmes will then employ various digital image processing techniques to extract the required information and make the decision of what to do with the image, usually by passing it on to the next part of the process, diverting it to another process, or discarding the item from the production or distribution system.

Machine vision equipment

Depending on the role of machine vision in the system and what the application of the technology is designed to be a solution to will change what type of equipment the vision system will need. Some systems will only need a few parts while others will require multiples of the same part across the processing line or solution system.

For example, if the system requires an inspection when a product comes onto the process line, then needs the different labels checked halfway through the system, and when the product comes to the final stage it might need closer detection for specific defects, it’s going to have a lot more detail compared to an item that just needs a label to be checked by a standard camera.

However, most times the components of an automatic inspection and sorting system will usually include some form of lighting to make the object clear and visible, a camera to see the object, a processor, software to understand the processor, and an output device or human interface (HMI) screen to visualise the data.

Cameras

There is a range of different cameras available for a machine vision system with different interfaces, pixels, resolutions, and features. The cameras are the primary piece of equipment for inspecting the object or item in a machine vision system.

It might be that the system needs to use multiple cameras for a process which is referred to as dual cameras. This will mean there are multiple cameras for one specific point of inspection and checking to make sure an otherwise hidden part can be properly checked.

Smart cameras

A smart camera is required when a machine vision system needs to capture and extract application-specific information from an image. A smart camera is capable of generating descriptions and making decisions. A smart camera usually contains all necessary communication interfaces as well as being connectable to wifi or a server to easily transfer captured image data.

3D cameras

3D cameras enable the depth of an item to be displayed in an image to show different angles of the image and give an idea for the shape of an item. By using a 3D camera in the machine vision system it will allow different perspectives and depth perception.

Thermal imaging cameras

A thermal imaging camera is a type of thermographic camera that renders images through infrared radiation which shows areas of heat on the images.

Software

A machine vision system requires software to visualise the data and show what the cameras are looking at for operators to analyse and maintain the system as well as programming the functions of the hardware. There is different available software that can be matched for what the machine vision system needs to do and what data needs to be visualised to operators from the system.

Embedded systems

Embedded systems for machine vision, which is also known as an imaging computer, is a camera without housing or a frame that is directly connected to a processing board. This combines all parts under one single board computer. Due to the increasing amount of open-source libraries for machine learning and AI, more computer vision systems are being deployed as embedded systems or IoT devices.

Frame grabbers

This is an electronic device that captures individual digital still frames from an analog video signal or a digital video stream. It can be used as an add on to a machine vision system to capture specific frames to analyse from a quick moving system.

Illuminators

These add light to the system so the camera has enough light to capture the image. Depending on the detail required in the imaging will change what type of lighting is needed for the machine vision system to identify what it needs to.

Lenses

Lenses will determine what resolution the camera and machine vision system can capture images at. The higher the pixel of the camera, the higher resolution the lenses will need to be and the larger the camera will likely be.

High performance computing

It’s not always required to have a high-performance computing system. They will be required when dealing with high volumes of items with different details.

These enable the machine vision system to process the information at a quicker rate and will usually be required if the system is using a capability such as machine learning, deep learning, or neural networks.

Cabling

As with all technology and applications, a machine vision system requires power cables to go to the main power supply to work and connecting cables between user interface such as a laptop or data centre. However, machine vision systems are increasingly being able to be used by connecting to the cloud to move the data to a visualisation or data platform in real-time.

Label verification

Depending on the application the system is being used for will change what type of label verification is needed. For an industry like food and beverage, there will be much more in detail packaging checks such as what the food is and what the sell by date is.

A label verification unit can send alerts, stop a moving system, verify an item, reject items from the process, and qualify alerts that are raised. Depending on the level of detail needed will change what label verification system is needed. A camera with machine vision capabilities is built into a label verification unit. They are popular in production lines and quality control systems.

Robots

Robots are increasingly having machine vision built into them to improve efficiency and accuracy. It enables robots to carry out more complex tasks that wouldn’t be possible without the system telling the robot exactly where to place the item.

Machine vision imaging process

The device used to capture the imaging (a camera of some sort discussed above) will either be separate from the main image processing unit or combined with it which will be a combination of a smart camera and smart sensor.

When the full processing function happens within the same enclosure this is using embedded processing and when it’s separated it will be processed in another piece of hardware or sent to a frame grabber within a computer using a visualisation tool such as Camera Link or CoaXPress to make sense of the imagery.

It’s also possible to use digital cameras capable of direct connections without a frame grabber to a computer via FireWire, USB, or Gigabit Ethernet.

Usually, a machine vision system uses conventional 2D imaging in standard lighting situations. However, when an object requires specific lighting to note the defects for example a machine vision system can use multispectral imaging, hyperspectral imaging, infrared bands, line scan imaging, 3D imaging, and X-ray imaging.

The key difference is that the usual 2D visible lighting images are monochromatic compared to more complex lighting which considers factors such as color, frame rate, resolution, and whether or not the imaging process is simultaneous over the entire image, making it suitable for systems that require the technology to track specific moving items.

Image processing

Once the image is captured it will go on to be processed usually by a central processing unit which are generally done by a CPU, GPU, a FPGA, or a combination of these three. Depending on the size of the system the machine vision equipment is being put on will change what processing unit is needed.

For example, if it’s a small operation that only needs to check a dozen items a day compared to a larger system that needs to check a dozen items a minute, the processing unit will likely vary in terms of what’s needed to process the volume of data. If the machine vision system was to implement machine and deep learning training, it can take higher-spec processing and performance requirements.

Image processing takes part in a sequence that gives the finished result for what that the machine vision system is being used to display to the end-user.

A typical sequence will usually start with tools such as filters which modify the image, then the objects or items will have their shape and details extracted, followed by the specific data that needs to be extracted from that shape or item (e.g. barcode, measurements, destinations or postcodes), this data will then be communicated to the processing unit to show where the item needs to be sent or if the item has passed/failed inspections, etc.

There is a range of both filters for the machine vision system to use and image processing methods to be applied. Indeed, depending on what the machine vision is being used for will affect what filters and image processing methods are going to need to be used.

Stitching/Registration

This is the process where images from different image capturing cameras are combined. In most cases, it will be the combining of a 2D camera with a 3D camera.

Filtering

Filtering allows for the machine vision system to appropriately categorise the image and ‘filter’ the image to a category that matches the suitable tags. The more complex the items going through the machine vision system, the more likely there will be increased filters for the item to meet.

The filters are similar to measurement or gauging in the sense that aspects like shape, size, material, and weight could be filters for the machine vision to inspect and consider.

Thresholding

This process enables the separation of images in portions. This requires the system to set a grey value for the images for what section to separate. Thresholding can be used to separate certain parts of the image into black and white that need to be identified as being different to the rest of the image.

Pixel counting

This process counts the number of light or dark pixels in the image and usually carried out by pixel counting sensors. A process this might be used is on a system for packaging where pixel counting sensors can detect bottle labels by combining the pixels to make up an image of what bottle it is.

Segmentation

This is the process of breaking up the digital image into multiple segments to simplify and/or change the representation of an image into something that is meaningful and easier to analyse what the object or item is or what category the object or item falls under.

Edge detection

Enables the machine vision to see edges of items that are coming through the process. Through machine learning, the technology can be taught how to spot different edges which might be required for sorting the items or objects on the process line.

Colour analysis

This allows the machine vision system to identify and assess quality of parts, products, and items using colour. In a similar way it can also isolate features, items, or products using colour to identify faults, changes, differences, or various defects.

Blob detection and extraction

This capability can detect an image for discrete blobs of connected pixels which allows the system to notice errors by spotting empty or different filled spaces to what should be there.

Machine learning, deep learning, and neural networks

Three technologies that can be used for processing items through a machine vision system at a quicker rate are machine learning, deep learning, and neural networks.

These three technologies can be applied to expand the machine vision understanding of what to spot making them a valuable asset to the machine vision if there are complex and various objects or items going through machine vision inspection.

Pattern recognition

By using pattern recognition the machine vision system can find, match, and/or count specific patterns that appear on the process line. Different patterns or more complex patterns to watch out for can be taught to the machine through machine or deep learning. Examples of this might be objects or items that may be rotated, partially hidden by another object, or varying in size.

Data reading

This is the ability to read the data on the labels or on the items usually through a data matrix (QR code), barcodes, or radio-frequency identification (RFID). Examples are being a barcode for specific items labeling what piece of clothing it is or it will be to read a barcode showing what part of the country or which country the item needs to be sent to for further sorting.

The level of data reading a machine vision system can do varies and can become more complex especially when using RFID scanning due to the various tags that can be used for different items.

Character recognition

Similar to reading data this allows the machine vision system to read text and numbers such as tags, categories, or serial numbers. The more complex the text becomes, the more likely you will need to upgrade your machine vision capabilities via training the system with machine or deep learning.

Measurement or gauging

Enables the capability of measuring items or objects dimensions during the imaging processing stage. This allows the vision system to understand the dimensions of the object in various measurement units such as pixels, inches, millimeters, length, time, weight, etc.

Decision result

This is what the object or item going through the system will be measured against to see where it goes in the process. For example, if a product doesn’t meet the required standards it will be pushed onto a different section compared to if it passes the standards where it will go to another section. Or it might be that if the item is going to a certain destination the machine vision system will pick that up through reading a barcode and send it on the route correct for that item.

It can also be used to spot defects on items to determine if it passes or fails the requirements needed to go on to the next section of the system. The more complex the defects that the machine vision system has to spot, the more likely the machine vision system will need machine and/or deep learning to spot the defects, similar to pattern recognition.

Outputs

In the process of inspection and sorting that can be done with machine vision, the output is the end section that gives decisions on what to do with the item, object, or product on the system. This is similar to the decision result except this is for the end of the system line.

It’s common for outputs to trigger mechanisms that reject failed items or objects while others might sound alarms to draw operators to a problem with a certain product or the entire system. For example the processing line might get full and cause a blockage that stops items flowing through machine vision which triggers an alert because the machine hasn’t inspected an item for a certain amount of time.

The data from an output can be recorded and interpreted to operators via a visualition dashboard or HMI to identify opportunities and possible methods of optimising either sections of the system or the entire system. This of course lowers risk of situations like the example with blockages happening again on the line which in turn increases efficiency.

Robotic guidance with machine vision

Robots are becoming increasingly used in the world today and as a result machine vision is becoming increasingly used with robots. When robots are equipped with machine vision it gives them much greater accuracy, orientation, and understanding. It allows them to more accurately grasp an item, place an item with increased precision, and carry out more complex tasks such as inspection in quicker time. This capability allows operators to move robots in a more simple manner by using a 1 or 2 axis motion controller.

In most cases a camera attached to a robot or a dual-system camera can be accurate for applications such as product sorting or robotic pick and placing. However, laser scanning is also another method that can be used to shine a striped light over the part that needs to be examined. These are similar to the way 3D cameras are used as they create data points to generate a 3D map of the selected item or part of the item.

The steps for machine vision included with a robot are the same as being in a fixed position on a process line as mentioned above with the differences being there is no focus on providing position and/or orientation information.

Machine vision market

Markets and Markets reported that the market size for machine vision is expected to grow from $10.7bn in 2020 to $14.7bn by 2025 at a compounded annual growth rate of 6.5%.

Machine vision is a capability and not an industry as are other capabilities such as automation, machine learning, deep learning, and neural networks. It’s a capability that can be integrated into other technologies and processes to benefit industries and improve business efficiency.

The machine vision market is growing and is not set to slow down anytime soon as businesses are increasingly looking to automate business processes through the introduction of automation capabilities like machine vision and robotics.

Due to COVID-19, businesses within logistics, healthcare, manufacturing, and food and beverage have been hit hard with the social distancing measures being brought in to stop the spread of the virus. Automation and robots for this reason are really showing their strengths and capabilities as businesses struggle to meet business and consumer demands with reduced labor.