Internet bots are web robots run by software based on performing and completing automated tasks online. Some of the tasks are operating search engines, power Application Programming Interface (API’s), vulnerability scanning, and monitoring website performances. Bots can also be used to perpetuate distributed denial of service (DDoS) attacks or take control of someone’s computer with malware.

The purpose of bots is to perform and complete online tasks faster than humans. There are multiple kinds of bots. The most common are monitoring bots, search engine bots, or crawler bots, and feed fetcher bots. Monitoring bots observe the website and identify areas that can continuously be improved. This can avoid unwanted bottlenecks and inefficiencies. Crawler bots download an index’s content from all over the internet. Once the information is collected, the bot will apply a search algorithm to index the data. Google has only indexed .004% of the internet. There are billions and even trillions of web pages on the internet. This can be applied to when a user is typing in the search bar and different suggestions appear. These suggestions can help speed up the process of getting to a specific website more efficiently. Feed fetcher bots are programmed to retrieve data to be displayed on a website. These bots retrieve information for websites and mobile applications and display the data.

As internet use increases worldwide, more users have access to create different automated pieces of software to conduct specified tasks. In 2020, about 37.2% of all internet users were bots. Of the 37.2% internet bots, they are broken down into two categories: beneficial bots and Malicious bots.

Google and Bing are among a massive list of companies that employ bots. For example, the Googlebot has two different functions. The first function is a search through each website on a desktop and index the results. This same method also applies to mobile sites. Googlebot Desktop and mobile should not visit your site more than once every few seconds on average. “Googlebot was designed to be run simultaneously by thousands of machines to improve performance and scale as the web grows.” Facebook is another large company that utilizes bots. One of their bots takes pasted links in posts and retrieves thumbnail pictures. This could be the title page of a website or the embedded tag of a video. Bing deployed a bot called Bingbot in 2010 that replaced the MSN bot. The bot supplies information to the bing search engine and indexes the information like a web crawler bot.

Figure 1: Bots and Company Usage (Accessible in the PDF Version)  

As of spring 2017, Facebook claims about 100,000 bots on their Messenger app while Twitter has 48 million bot accounts. Kik claims 20,000 bots on its platform and the Microsoft Bot framework claims more than 20,000 developers have signed up for bots.

Create a Bot

There are two options to obtaining an internet bot. The first option is to download or use a program that allows you to create and modify a bot. The bot is already coded with the proper instructions to run. One program is airSlatre. This service allows you to create a bot with automated tasks, schedule tasks for the bot to perform, and requires no coding to configure the bot. The functions of the bot can be specified once the program has been selected. For example, you can customize a chatbot and its output. Chatbots are implemented into Discord, Facebook Messenger, and many other messaging applications. Once you have customized the functionality of the bot, then you can deploy it. Bots can be obtained by downloading them. Many bots are fully functional and available for public use. Examples of downloadable bots are the IBM Watson Assistant, ManyChat, Amazon Lex, and Google Cloud Dialogflow. Some bots are free for the public to use while others, for instance, Google’s Cloud Dialogflow require getting a quote before obtaining it.

The second option is to create an internet bot from scratch. This requires a software engineering team that specializes in AI and bot creation.

The process is broken down into different parts.

1. Find a Purpose: Identify opportunities for an AI-based bot (Kumar, 2017). Questions to answer are what kind of bot does your company need? What is the goal of the bot?
2. Conceptualize the flow: Design the functionality of the bot. What specific tasks is it going to perform? If it’s a chatbot, what are some of the automated messages it will output?
3. Design the Bot: Design the non-coding frameworks and the coding frameworks of the bot (Kumar, 2017). This is where the bot starts to come together. Implement the AI part of the bot to allow it to run on its own.
4. Use Prototyping Tools: This is the testing phase where the bugs can be identified and resolved before the initial launch (Kumar, 2017). Make sure the bot meets the expectations of the initial goal and can perform the tasks that it is responsible for.

Malicious Bots

Malicious bots have been around almost as long as computers have been around. Malicious bots conduct illegal activities to gain access to private information, scam users or temporarily shut down business servers. Malicious bots perform Distributed Denial of Service (DDoS) attacks, Web scraping, Spamming, Impersonations, and Hacking. One of the most visible DDoS attacks in 2017 was an attack on Google. A Chinese-backed hacking group conducted a DDoS attack on Google that flooded the internet traffic (Afifi-Sabet, 2020). This lasted for about six months and peaked at 2.5 Terabytes per second (Tbps). This surpassed the 2.3 Tbps attack on Amazon Web Services in 2020. The target of the attack was not disclosed.

Impersonator bots impersonate people on the web to gain access to private information. Impersonator bots often target online bank accounts for valuable information. Social security numbers, account numbers, or any personally identifying information. Many of the online services have fraudulent protection services to detect impersonating bots for stealing valuable data. Other malicious bots log keystrokes, relay spam, and capture and analyze internet packets, which are small segments of a larger message sent over the internet and are combined by the receiving computer. Logging keystrokes allows a hacker to access usernames and passwords are for a specific account. Many users use the same username or password for different accounts. 31.3% of people change their password once or twice a year. 22.4% change their password more than five times a year and 17% change their password every few months. 29.4% of people rarely change their passwords while only 10.9% of people never change their passwords. This makes it easier for criminals to access all of your accounts.

How to Protect from Malicious Bots

Malicious bots pose a threat to disclose private information. Businesses are increasing their safety measures to protect their data, websites, and customers. One small way you can protect yourself from malicious bots is to always stay up to date with the latest version of any given software. All of the company computers should be on the same platform and update around the same time.

This is to ensure that no computer is left vulnerable and make it an easy target for hackers and bots. As computers and software evolve, bots become more and more difficult to detect. Businesses can protect themselves from malicious bots by integrating a web security scanner into their systems. This will allow you to have an end-to-end view of your most vulnerable points. One technology has emerged to isolate any IP address showing aggressive or unusual behavior. This can lead to the early detection of malicious bots before they enter your website. There are other bot mitigation services available that give you full control over the wide range of bots that access your website every day. With this technology, they can identify outdated users. These user agents can be blocked or caught by a Completely Automated Public Turing test to tell Computers and Humans Apart (CAPTCHA). CAPTCHA is a program or system intended to distinguish humans from machine input. One example of CAPTCHA is the text-based system. This is shown on websites when it prompts the user to verify if they are human. The user is prompted to enter a series of characters that range from lowercase and uppercase letters and numbers.

Figure 2: Human Verification (Accessible in PDF Version) 

This is one of the more common CAPTCHA services. Another variation is CAPTCHA Image. This will prompt the user to select all of the images that are of an object or animal. This could be a dog, cat, a bus, a traffic light, street sign, etc. Figure 3 is an example of a CAPTCHA Image. This example shows that the user has to match all of the images to the one in the top right. No program or software is perfect to protect from malicious bots. Some of the drawbacks of using CAPTCHA are that it is disruptive and frustrating for users, can be difficult to understand, some CAPTCHA types are not supported on all browsers, and some CAPTCHA types are not accessible to users who view a website using screen readers or assistive devices (Imperva). CAPTCHA is not the only solution to protecting from malicious bots. It is only part of the solution. Other practices need to be implemented to ensure the safety of the website and the data. Companies can host training programs and security awareness, block known hosting providers and proxy services, protect every access point, carefully evaluate traffic sources and failed login attempts, and investigate traffic spikes.

The Future of Bots

The future of bots is growing rapidly. As technology advances, bots will advance with new tasks to perform and different functionalities. The mark size in 2019 was $2.6 billion and looks to grow to 9.6 billion in 2024 (“The Future of Bots”). The future of bots will continue to grow as computers become more and more a part of our daily lives. Humans will learn to adapt to internet bots and the growing functionality. Chatbots are becoming more and more personalized to better user experiences. The added personalization stems from the combination of Natural Language Processing (NPL) and Machine Learning practices. Both of these technologies automate tasks previously done by humans and allows bots to evolve from a menu-based approach to a more human-like conversational and user intent approach. Many researchers are developing and improving the AI and learning capabilities in chatbots. Chatbots are now moving into voice assistants, Siri and Amazon Alexa.

Siri has been around since late 2011. Each year, these services become more advanced and introduce new features to make the user experience better. Both of these voice assistants allow you to control objects in a household, purchase from the developer’s website, and function as a speaker. This is only a shortlist of functions of Siri and Amazon Alexa. In 2019, 3.25 billion voice assistants were being used. This number increased to 4.2 billion in 2020. It is projected that in 2024, 8.4 billion voice assistants will be used. The figure below clearly shows the 5 most popular chatbot builders in 2020.

Figure 3: Most Popular Chatbots (Accessible in PDF Version) 

As a result of the diagram, ManyChat has about a search volume of 318,930. Dialogflow has about 82,640, Chatfuel has 79,130, MobileMonkey has 21,100, and Haptik has 20,610. In total, this adds up to 522,410 search volume per month.

Conclusion

In conclusion, developers are designing new bots with more advanced AI and Machine Learning. This allows bots to process information independently. This applies to both beneficial bots and malicious bots. Many precautions are being taken by businesses to protect themselves from these bots. Over the past year, companies are using software services to create bots specific to the needs of the business, for instance, protection, and accessibility to confidential information. Bots will become more accessible in the future and be implemented in different ways applicable to the goals of the organization.