Embedding safety in internal policies and procedures

Good safety review processes should include testing: 

• specific edge cases, such as unusual user behaviour or incidents that require special handling
• across all channels, surfaces, features and tools
• with diverse teams, incorporating members from different genders, backgrounds, experiences and perspectives
• within specific regions and jurisdictions.

Good safety review processes should include analysis of:

• patterns of behaviour and social network effects, focusing on abusive actors
• online signals such as metadata and traffic signals (such as notifications, reaction buttons, read receipts, active status)
• behavioural signals including patterns of interactions, such as search activity, group membership and activity, violation indicators (such as reports and connection activity or friend requests), content creation and sharing, profile and accounts
• behavioural signals for at-risk groups (such as engagement patterns, content preferences, use of reporting and content control features)
• interactions between/across different parts of the product, platform or service.

The environmental context your platform or service operates in constantly changes so safety review processes require:

• continuous assessment of new forms or techniques of abuse occurring on the platform
• researching and analysing new forms or techniques of abuse on other platforms
• information sharing cross-industry including on good practices and new safety innovations
• understanding the support requirements and resources for victims/survivors and at-risk and marginalised groups.
   

Good safety review processes should assess the effectiveness, accuracy and impact of: 

• automated and human moderation systems (such as removing, demoting, delaying and/or labelling of content, and suspension or banning of accounts)
• user safety controls and tools (such as muting and blocking)
• safety and civility campaigns (such as encouraging respectful behaviour and educating users on community guidelines)
• prevention interventions (such as behavioural nudges, warning labels, parental controls)
• reporting and appeal systems and processes
• disruption techniques (such as geoblocking, content throttling)
• detection tools
• automated responses
• feedback systems and processes
• the reduction of harms or risks, with a focus on at-risk or marginalised groups
• any user confusion or misunderstanding relating to how a product or feature functions.
• selected data for building AI models.

You should also assess Health and wellbeing impacts and provide the necessary support for all stakeholders involved in managing harms and risks, including: 

• internal and external employees involved with addressing harms and risks
• community managers or community moderators or administrators. 

Prior to launch, your product, platform or service should be tested again. This may take the form of ‘beta testing’, where a select group of external users conduct real-world testing, or ‘dogfooding’, where employees use the product or service before its released to the public.

Testing your product should help you identify and mitigate pitfalls and emergent risks that you may not have considered in your safety scenario testing, analysis or assessment.

To ensure objectivity and fresh perspectives, safety measures, such as policies and tools, should be reviewed by independent third-party auditors. Seeking out innovations and research can also improve safety review processes.

Your safety review should be informed by national and international: 

• regulatory frameworks, including requirements under the Online Safety Act
• technical standards, such as those from the International Organization for Standardization. 

Establish a baseline risk profile for users, especially those with connected accounts across multiple platforms, by considering:

• basic profile information: username, display name and associated URLs.
• demographic attributes: gender, age, religion, ethnicity, ability, sexual orientation and political affiliation.

Evaluate the types of content a user interacts with and assess them for potential risk based on context, frequency and sentiment. This includes: 

• images
• text
• audio
• video.

Use detector presets that are linked to a library of keywords and phrases and monitored in real-time. These flag high-risk behaviours in user interactions, including:

• sexual harassment/aggression
• profanity
• nudity
• doxing
• sharing prohibited content
• ideological extremism
• threats of violence
• impersonation
• insults
• harassment
• self-harm
• defamation.

Apply analytical tools, including:

• account behaviour analytics, to track changes in user activity over time
• sentiment analysis, which evaluates tone and emotional content
• linguistic patterns, to detect patterns that may indicate escalating risk.

The risk detection process involves:

• flagging prohibited activities or interactions that have a risk indicators
• quickly filtering through data to locate relevant risk thresholds
• establishing a risk score
• linking prone patterns into analysed behavioural changes over time
• establish links to other profiles with similar risk scores.

Beyond individual behaviour, consider external and behavioural factors, such as:

• group memberships
• vocabulary and language use
• social connections and interaction strength
• environment risk profiles
• romantic relationships
• geospatial analysis, locations and movements
• page likes and follows
• interactions with forums
• influence and prominence in networks
• online signal analytics
• events interacted with
• network analysis.

Ensure you understand the typology of harms that may occur and the tactics and techniques used by perpetrators. Learn about online harms and common misuses of online tools and features.

Assess and document all impacts on user safety – positive, negative and differential. Impacts could include: 

• short, medium and longer-term impacts
• actual and potential impacts
• differential levels of impact among at-risk, vulnerable and/or marginalised groups
• impacts on the target audiences or userbase, and those who may be indirectly impacted
• how distinct or interrelated the safety impacts are, considering increases in effects and impacts
• whether the safety impacts are caused, contributed to or directly linked to the platform’s design, development or deployment. 

Assess the severity of consequences based on: 

• the scope, scale and interrelatedness of impacts
• whether impacts can be remedied
• the consequences for individuals and society. 

Make the following information available to all employees: 

• Corporate policies and procedures such as accountable authority instructions, risk management guides and policies covering compliance and enforcement.
• The code of conduct and its relation to user safety.
• The platform’s terms of service or community standards and guidelines.
• The types of online harms and current techniques and tactics used by abusers.
• Mitigation measures such as the technical features, advice and support available.
• What actions are taken when violations occur.

Leaders and human resources teams may require additional details on: 

• role-specific responsibilities and expectations in terms of managing user safety
• access levels to internal tools and user account information based on an employee’s role
• policies on the appropriate use of resources such as systems, hardware and data
• the types of internal safety breaches that may occur, including edge-case examples. 

Ensure all staff have access to clear safety governance processes and procedures, including: 

• safety review cycles and teams involved
• internal and external safety intelligence sharing procedures
• reporting, complaint, appeal and escalation processes.  

Leaders and human resources staff should also have access to the policies and procedures for: 

• enforcement of privacy, security and safety compliance
• identifying, moderating and reporting illegal or harmful activity carried out by employees or using corporate equipment
• procedures for handling employee non-compliance or incorrect/incomplete reporting
• dispute resolution, review and appeal processes for staff misconduct. 

An acceptable use policy outlines the practices and constraints a user must agree to before they use or access a platform or service. It is crucial to have training and awareness of the policy at your organisation. It should be implemented from the earliest stages possible for all employees, including contractors.

Promote the policy through:

• the hiring process and employment contracts
• induction and ongoing training
• team meetings
• intranet and email updates.

Support staff to understand the policy by:

• writing in accessible and clear language
• using case studies or examples
• providing practical video or illustrations
• requiring written confirmation from staff
• tracking or monitoring online access to the policy
• including compliance with the policy in performance assessments.

Learn more about acceptable use policies in the module Dealing with illegal and restricted online content.