We may ask AI to summarise a report, review an email, analyse customer data, translate a document, or offer advice on a personal problem. These features can save considerable time. At the same time, however, the information we enter may include names, email addresses, telephone numbers, internal documents, health records, financial information, or other details that should not be disclosed.

The important question, therefore, is not only what AI can do for us, but also what information we are giving it in exchange for that convenience.

How Is AI Changing Data-Related Risks?

AI did not create all the privacy problems we face today. Excessive data collection, unauthorised access, and data leaks existed long before the arrival of generative AI.

What AI changes is the scale, speed, and ability to connect information. AI systems can process large quantities of data, detect patterns that humans may overlook, and infer sensitive information from details that appear insignificant. When AI is connected to email accounts, databases, internal documents, or organisational systems, the consequences of an error or attack can become much more extensive.

The Office of the Australian Information Commissioner notes that generative AI presents particular privacy risks because of its probabilistic nature, reliance on large amounts of training data, and vulnerability to malicious use. However, not every AI system carries the same level of risk. Much depends on how the system is designed, what type of data it processes, and how it is used.

1. Collecting More Data Than Necessary

The development of AI, particularly large language models, often relies on information gathered from many sources, including websites, books, articles, databases, and specially prepared datasets. This raises important questions. Do the people whose information appears in those sources know that it is being used? Did they give consent, and if so, what exactly did that consent cover?

For organisations, another risk arises from the assumption that giving AI more data will always produce better results. This may encourage people to upload entire datasets, even when only a small portion is needed for the task.

A safer approach is data minimisation: using only the information necessary for a specific purpose. If an AI system is being asked to identify sales trends, it may not need the names, addresses, or telephone numbers of individual customers. If the task is to improve the language of a document, there may be no reason to upload the full report together with confidential organisational information.

2. Data Leaks Caused by What We Enter Ourselves

The most immediate risk may not come from a sophisticated hacker or a complex technical vulnerability. It may begin when someone copies information into an AI chatbot without stopping to consider what that information contains.

An employee might ask AI to summarise an unpublished contract. A lecturer might upload student information to help classify assessment results. A staff member might enter a client’s medical report, while an individual user might upload an identity document and ask AI to read it.

The purpose may be entirely innocent, but users should still ask whether they are authorised to remove that information from its original system and submit it to an external AI service. They should also understand how the service stores, processes, and deletes information.

The fact that a tool is “free” or accessed through a personal account does not make it a private space. This distinction becomes especially important when public AI tools are used to handle information belonging to a university, company, government agency, customer, student, or patient.

3. Models May Memorise Some Information

AI models do not normally function like databases that search for and retrieve an original document word for word. Nevertheless, security research has shown that some models may memorise portions of their training data, particularly unusual information or text that appears infrequently.

Under certain conditions, attackers may submit repeated or specially designed queries in an attempt to extract information associated with the training data. Such attacks might seek to determine whether a particular person’s information appeared in a dataset or attempt to reconstruct parts of the original data.

This issue must be explained carefully. It does not mean that every AI system will routinely memorise and reveal personal information during an ordinary conversation. The level of risk depends on the model, its training data, the training process, the provider’s safeguards, and the attacker’s capabilities.

For this reason, the NIST report on Adversarial Machine Learning treats memorisation, membership inference, and data reconstruction as risks that require serious assessment.

4. Removing a Name Does Not Always Make Someone Anonymous

Many organisations assume that deleting names from a dataset is enough to prevent individuals from being identified. In reality, AI can connect details from multiple sources.

A dataset may contain no names but still include a person’s age, occupation, area of residence, and date of receiving a particular service. When those details are combined with publicly available information or another dataset, they may be sufficient to identify the person concerned.

This risk is known as re-identification or de-anonymisation: the process of linking apparently anonymous information back to a real person. NIST warns that AI systems can create new privacy risks by inferring a person’s identity—or previously private information about that person—from data scattered across different sources.

Removing names and identification numbers may therefore be insufficient. Organisations must also consider whether the remaining details, when combined, could still point to a particular individual.

5. When Malicious Instructions Are Hidden in a Document

Prompt injection involves placing instructions in content with the intention of making an AI system disregard its original rules or behave in a way its developers did not intend. These instructions may come directly from a user, but they can also be hidden in websites, emails, or documents that an AI system has been asked to read.

Imagine an AI assistant that reads incoming emails and prepares a daily summary. One email contains a concealed instruction telling the system to search for confidential documents and transmit the information elsewhere. If the AI has been given excessive access and the surrounding system lacks adequate controls, it may be manipulated into performing actions it should never have been allowed to perform.

Prompt injection becomes more serious when AI does more than answer questions—when it can open files, search databases, send messages, or take actions on a user’s behalf. OWASP identifies prompt injection as one of the major security risks facing applications built around large language models.

The problem is not simply that an AI system may “disobey” its instructions. The more important question is how much authority the surrounding system has given it.

Does Turning Off Model Training Make Our Data Safe?

Many AI services allow users to prevent their conversations from being used to train or improve future models. Turning off this option is a sensible step, particularly when using a personal account.

However, disabling model training is not the same as preventing all data storage. A service may still retain conversations for a specified period to maintain chat history, detect misuse, meet legal obligations, or protect the security of the system. Policies may also differ across personal accounts, organisational accounts, business products, APIs, and different service providers.

OpenAI, for example, states that users can turn off “Improve the model for everyone” to prevent new conversations from being used for model training. Those conversations may still remain in the user’s chat history. Temporary Chat, by contrast, is not used to train models and follows a separate deletion process. Further details are available through OpenAI Data Controls.

Users should therefore look beyond a single settings switch. They should examine how a service handles data retention, deletion, access, security monitoring, and model training as separate issues.

What Should We Do Before Sending Information to AI?

Security does not depend entirely on the safeguards provided by an AI company. It also begins with the decisions users make before pressing “Send” or uploading a file.

• Do not enter information that you are not authorised to disclose. This applies even when AI is being used only to summarise, translate, or proofread a document. If the information belongs to an organisation, customer, student, patient, or another person, users should confirm that they have permission to remove it from its original system and submit it to an AI service.

• Provide only the information needed for the task. If AI is required to edit a single paragraph, there is no reason to upload an entire document. Data minimisation reduces the potential harm if the system makes an error or the information is exposed.

• Remove or replace personally identifiable information. This may include names, email addresses, telephone numbers, identification numbers, health information, financial details, and confidential business information. Users must also consider whether the remaining details could still be combined to identify the person, since removing a name alone may not be enough.

• Use services approved by the organisation. Users should check how long the provider retains information, who can access it, whether users can delete it, and whether it may be used to train or improve a model. Personal accounts, organisational services, and API-based products may operate under different conditions.

• Disable model training when the option is available. This reduces one form of data use, but it does not necessarily prevent the service from storing the information. ChatGPT, for example, separates controls over model improvement from chat-history retention. Users should review both through OpenAI Data Controls.

• Give AI only the access it needs. Organisations should follow the principle of least privilege. If an AI system only needs to read documents, it should receive read-only access to the relevant files rather than unrestricted access to the entire database. These controls should be supported by multi-factor authentication, encryption, activity logs, and regular security reviews.

• Require human approval for consequential actions. Human authorisation should remain necessary when AI can send emails, disclose documents, alter records, conduct transactions, or communicate with people outside the organisation. Systems should not be allowed to perform high-impact actions entirely on their own.

• Assess privacy before a project begins. Organisations should decide from the outset what information is genuinely necessary, who could be harmed if it were exposed, and when it should be deleted. This reflects the principle of privacy by design: preventing privacy problems during the design stage rather than responding only after a breach has occurred, as recommended by the UK Information Commissioner’s Office.

These practices cannot eliminate every risk. They can, however, help us make informed decisions before submitting information, rather than asking questions only after the data has moved beyond our control.

We Do Not Need to Stop Using AI—but We Do Need Boundaries

AI offers significant benefits for education, employment, research, and creative work. The issue is not simply a choice between using AI and rejecting it. What matters is how well we understand the system we are using and whether we recognise whose information we are involving.

Before sending a message or uploading a document, we can pause for a few seconds and ask: Who does this information belong to? Am I authorised to use it? Is it necessary to submit all of it? Who could be affected if it were disclosed?

Privacy in the age of AI is not about hiding from technology. It is about retaining the ability to decide what information may be used, for what purpose, by whom, and for how long.

We do not need to fear that AI may come to know us. But we do need to recognise when it is beginning to know too much.