Why Older Adults Are at Greater...
Impersonation scams are making it easier for fraudsters to exploit trust, particularly where older relatives or vulnerable customers are involved....
Read More
Artificial intelligence is increasingly being used to make everyday business processes faster and more efficient. From reading and summarising emails to booking meetings, querying documents and supporting customer service, AI tools are quickly becoming part of the modern workplace.
However, as AI assistants and chatbots are given more access and more responsibility, organisations need to understand the risks as well as the benefits. When an AI tool can access mailboxes, file shares, customer records or account recovery processes, it can become more than a helpful assistant. It can become a new route into sensitive systems and data.
This is particularly important when AI is trusted to act on behalf of a user or organisation. If the right controls are not in place, attackers may be able to manipulate the system, influence its responses or persuade it to take actions that should require stronger verification.
AI chatbots are designed to respond quickly, follow instructions and provide helpful answers. In many settings, that can bring real value. They can reduce administrative burden, improve response times and help users access information more easily.
The problem is that helpfulness can also create risk.
Unlike a trained member of staff, an AI chatbot may not always have the judgement, context or scepticism needed to recognise when a request is suspicious. It may provide inconsistent answers to similar questions, misinterpret information, or respond to prompts in ways that were not anticipated by the organisation that deployed it.
Where AI tools are connected to sensitive systems, the risk increases further. A chatbot that can read documents, access inboxes, retrieve customer information or make changes to an account needs to be governed carefully. It should not be treated simply as a convenience tool. It needs appropriate access controls, monitoring, testing and escalation routes.
A recent incident involving Instagram has highlighted the issue clearly. Hackers reportedly tricked Instagram’s AI support tool into helping them gain access to other users’ accounts.
According to reports, attackers were able to fake their location and ask the AI support assistant to change the email address associated with an account. Once the email address had been changed, they could then reset the password and take control of the account.
Meta said the issue had been resolved and that impacted accounts were being secured. However, the incident raises wider questions about how AI tools are used in sensitive workflows, particularly where they are involved in identity verification, account recovery or access changes.
The key issue is not simply that an AI chatbot made a mistake. It is that the chatbot appears to have been given the ability to carry out an action that had significant security consequences.
AI can support decision-making, but it should not automatically replace human judgement in high-risk situations.
Account recovery, identity checks, access changes and data disclosure requests all involve sensitive decisions. In these areas, convenience must be balanced against security. If an AI tool is able to approve changes without sufficient verification, it can be manipulated in much the same way as a person might be targeted through social engineering.
The difference is scale. An AI tool may be available at all times, handle large numbers of requests and apply the same flawed logic repeatedly if the underlying controls are weak. This makes it essential that organisations understand exactly what their AI tools are allowed to do, what systems they can access and where human oversight is required.
One of the biggest challenges with AI assistants is that they often operate with delegated authority. If a bot is connected to a user’s mailbox, files or account settings, it may be able to take actions in that user’s name.
That creates important questions for organisations:
These questions need to be answered before AI systems are embedded into business-critical processes.
Organisations using AI chatbots or AI assistants should take a structured approach to security and governance. This means looking beyond the technology itself and considering how the tool fits into wider risk management, access control and data protection frameworks.
Practical steps include:
AI tools can be valuable, but they need to operate within clear boundaries. The more authority they are given, the stronger the governance and oversight need to be.
At AJC, we help organisations understand and manage the risks created by new technology, including AI-enabled tools, automated processes and the wider governance structures that sit around them.
Our AI Risk and Governance Services support organisations in reviewing how AI systems are being used, what data and systems they can access, and where there may be gaps in governance, cyber security, compliance or operational resilience. This can include assessing AI and cyber security frameworks, reviewing threat exposure and vulnerabilities, and evaluating incident response and business continuity arrangements.
We can also help organisations consider whether AI tools are being used safely in sensitive workflows, such as customer support, account management, identity verification or access control. This includes looking at whether appropriate controls, oversight, escalation routes and reporting are in place.
As AI becomes more embedded in day-to-day operations, organisations need confidence that these tools are not creating avoidable security, data protection or governance risks. AJC provides practical, proportionate support to help strengthen oversight, identify areas for improvement and align AI adoption with wider risk management objectives.
AI chatbots and assistants are becoming increasingly common, but they should not be treated as risk-free.
When AI is given access to systems, data or decision-making processes, it needs to be governed with the same care as any other business-critical tool. Without clear controls, AI can move from being a helpful assistant to a potential security weakness.
The Instagram incident is a useful reminder that AI security is not just about what a tool says. It is also about what it is allowed to do.
Contact us on 020 7101 4861 or email us at info@ajollyconsulting.co.uk if you think we can help.
Sources:
BBC News. Instagram says it has resolved an issue which saw hackers trick its AI support tool into giving them access to other users’ accounts. Available at: https://www.bbc.co.uk/news/articles/c98rzr72dpyo
Business Insider. They tricked Meta’s chatbot to hack into Instagram accounts. It’s a warning about how much we trust AI. Available at: https://www.businessinsider.com/hackers-tricked-meta-chatbot-access-instagram-accounts-2026-6
404 Media. Hackers Simply Asked Meta AI to Give Them Access to High-Profile Instagram Accounts. It Worked. Available at: https://www.404media.co/hackers-simply-asked-meta-ai-to-give-them-access-to-high-profile-instagram-accounts-it-worked/
Image accreditation: Zyanya Citlalli (April 2023) from Unsplash.com+. Last accessed on 10 June 2026. Available at: https://unsplash.com/photos/a-small-robot-holding-a-laptop-with-a-speech-bubble-above-it-dIMLdS-or5s
Impersonation scams are making it easier for fraudsters to exploit trust, particularly where older relatives or vulnerable customers are involved....
Read More
In this series, we introduce the people behind AJC and the experience they bring to the business. In this profile,...
Read More
Cyber Essentials is now a recognised benchmark for UK businesses that want to demonstrate good cyber hygiene, reassure customers, meet...
Read More