Need help dealing with violent or distressing online content? Learn more

Safety by Design

 

Safety by Design

 

Safety by Design puts user safety and rights at the centre of the design and development of online products and services. 

Rather than retrofitting safeguards after an issue has occurred, Safety by Design focuses on the ways technology companies can minimise online threats by anticipating, detecting and eliminating online harms before they occur. It encourages technology companies to alter their design ethos from ‘moving fast and breaking things’ or ‘profit at all costs’ to ‘moving thoughtfully’, investing in risk mitigation at the front end and embedding user protections from the get-go. 

This proactive and preventative approach focuses on embedding safety into the culture and leadership of an organisation. It emphasises accountability and aims to foster more positive, civil and rewarding online experiences for everyone.

Safety by Design also acknowledges the need to make digital spaces safer and more inclusive to protect those most at risk.

All Safety by Design initiatives have been developed through in-depth research and consultation with industry for industry.

On this page:

Safety by Design principles

At the heart of Safety by Design are three principles that provide platforms and services with guidance as they incorporate, assess and enhance user safety.

  1. Service provider responsibility
  2. User empowerment and autonomy
  3. Transparency and accountability.


These principles outline realistic, actionable and achievable measures that providers of all sizes and stages of maturity can use to safeguard users from online risks and harms. They are built around a human-centric approach that places the safety and rights of users at its core, while also taking into account their needs and expectations. The principles elevate user safety as the third pillar in the developmental process for all online and digital technologies, sitting alongside privacy and security.

The principles also promote the technology industry’s strengths in innovation, encouraging new thinking and investment that supports product development which prioritises online safety.

The technology industry has a key role to play in ensuring these principles are adopted and their implementation is led from the top.

1. Service provider responsibility

The burden of safety should never fall solely upon the user. Every attempt must be made to ensure that online harms are understood, assessed and addressed in the design and provision of online platforms and services.

This involves assessing the potential risks of online interactions upfront and taking active steps to engineer out potential misuse, reducing people’s exposure to harms.

To help ensure that known and anticipated harms have been evaluated in the design and provision of an online platform or service, the following steps should be taken:

  1. Nominate individuals or teams and make them accountable for user safety policy creation, evaluation, implementation and operations.
  2. Develop community guidelines, terms of service and moderation procedures that are fairly and consistently implemented.
  3. Put in place infrastructure that supports internal and external triaging, clear escalation pathways and reporting on all user safety concerns, alongside readily accessible mechanisms for users to flag and report concerns and violations at the point they occur.
  4. Ensure there are clear internal protocols for engaging with law enforcement, support services and illegal content hotlines.
  5. Put processes in place to detect, surface, flag and remove illegal and harmful behaviour, contact and content with the aim of preventing harms before they occur.
  6. Prepare documented risk management and impact assessments to assess and remediate any potential online harms that could be enabled or facilitated by the product or service.
  7. Implement social contracts at the point of registration. These outline the duties and responsibilities of the service, user and third parties for the safety of all users.
  8. Consider security by design, privacy by design and user safety considerations which are balanced when securing the ongoing confidentiality, integrity, and availability of personal data and information.

 

2. User empowerment and autonomy

The dignity of users is of central importance. Products and services should align with the best interests of users.

This principle speaks to the dignity of users, and the need to design features and functionality that preserve fundamental consumer and human rights. This means understanding that abuse can be intersectional, impacting on a user in multiple ways for multiple reasons, and that technology can deepen societal inequalities. To combat this, platforms and services need to engage in meaningful consultation with diverse and at-risk groups, to ensure their features and functions are accessible to all.

To help ensure that features, functionality and an inclusive design approach give users a level of empowerment and autonomy that supports safe online interactions, the following steps should be taken:

  1. Provide technical measures and tools that adequately allow users to manage their own safety, and that are set to the most secure privacy and safety levels by default.
  2. Establish clear protocols and consequences for service violations that serve as meaningful deterrents and reflect the values and expectations of the users.
  3. Leverage the use of technical features to mitigate risks and harms, which can be flagged to users at relevant points in the service, and which prompt and optimise safer interactions.
  4. Provide built-in support functions and feedback loops for users that inform users on the status of their reports, what outcomes have been taken and offer an opportunity for appeal.
  5. Evaluate all design and function features to ensure that risk factors for all users – particularly for those with distinct characteristics and capabilities –have been mitigated before products or features are released to the public.

 

3. Transparency and accountability

Transparency and accountability are hallmarks of a robust approach to safety. They not only provide assurances that platforms and services are operating according to their published safety objectives, but also assist in educating and empowering users about steps they can take to address safety concerns.

The publication of information relating to how companies are enforcing their own policies and data on the efficacy of safety features or innovations will allow accurate assessment of what is working. If interventions are improving safety outcomes for users or deterring online abuse, these innovations should be shared and more widely adopted.

To enhance user trust, awareness and understanding of the importance of user safety, platforms and services should:

  1. Embed user safety considerations, training and practices into the roles, functions and working practices of all individuals who work with, for, or on behalf of the product or service. 
  2. Ensure that user safety policies, terms and conditions, community guidelines and processes about user safety are accessible, easy to find, regularly updated and easy to understand. Users should be periodically reminded of these policies and proactively notified of changes or updates through targeted in-service communications.
  3. Carry out open engagement with a wide userbase, including experts and key stakeholders, on the development, interpretation and application of safety standards and their effectiveness or appropriateness.
  4. Publish an annual assessment of reported abuses on the service, alongside the open publication of meaningful analysis of metrics such as abuse data and reports, the effectiveness of moderation efforts and the extent to which community guidelines and terms of service are being satisfied through enforcement metrics.
  5. Commit to consistently innovate and invest in safety-enhancing technologies on an ongoing basis and collaborate and share with others safety-enhancing tools, best practices, processes and technologies.

 

Audio

Hello, I'm Julie Inman Grant, and I am Australia's eSafety Commissioner.

eSafety is delighted to share its Safety by Design initiative with you.

eSafety is Australia's national online safety regulator. The first one of its kind. 

We lead, coordinate and advise on a range of safety issues to ensure that all Australians have safer and more positive experiences online.   

We approach our work through three different lenses: prevention, protection and proactive change, or the three 'p's.

It's the third 'p' I would like to discuss my talk on today: Proactive and systemic change.

If we're truly going to make the online world a safer place in the future, we've got to minimise the threat surface.

We must do better in terms of anticipating, detecting and eliminating the proliferation of online abuse.

We also need to make our digital spaces safer and more inclusive by protecting those most at risk.

So how do we go about affecting change right now? 

At eSafety we strongly believe that the answer lies in Safety by Design.

Safe design principles and standards are commonplace in the physical world, but this was not always the case.

It's hard to believe that it's been over 50 years since the campaign to put seatbelts in every car first began.

But now when we get into our cars, we almost take for granted that seatbelts, airbags, anti-lock brakes and other protections guided by international standards will help keep us safer on the road. Safety is built in by design.

But we know that the online world was not built for safety; it was built for speed.

We need to alter the design ethos that takes us from moving fast and breaking things, and profits at all costs, to one that focuses on moving thoughtfully, investing in risk mitigation at the front end and embedding user protections from the get-go.

The expectation of user safety should be as much a priority for technology companies as it is for the food, toy and automotive industries.

Prioritising the rights and dignity of users in product design and development will engender a culture of innovation, will motivate workforces, and ultimately will positively impact bottom lines. It just makes good business sense.

At eSafety, we've already been working with industry since 2018 on the Safety by Design initiative. 

We knew that to affect cultural change and for Safety by Design to be taken on and lead from the top, that industry genuinely needed to be at the heart of this process.

We also understood that we needed to play to industry strengths by encouraging both investment and innovation in product development and innovation that prioritises safety.

It was also vitally important that Safety by Design reflected the diverse needs of the technology ecosystem.

So we also consulted with NGOs, advocates, parents and young people themselves.

This truly collaborative effort yielded three overarching Safety by Design principles.

The first of these is 'Service provider responsibility'. 

The burden of safety should never fall solely upon the user and very attempt must be made to ensure online harms are understood, assessed and addressed in service design and provision.

This involves assessing the potential risks of online interactions upfront and taking active steps to engineer out potential misuse, reducing people's exposure to harms.

Indeed, it's really great to see many technology companies investing in and employing innovative hybrids of human moderation alongside advanced machine learning and AI tools, helping to minimise exposure to traumatic material.

The second Safety by Design principle is 'User empowerment and autonomy', which of course speaks to the dignity of users and the need to design features and functionality that preserve fundamental consumer and human rights.

This means understanding that abuse is intersectional and that technology can exacerbate societal inequalities.

To combat this, we need to engage in meaningful consultation with diverse and at-risk groups to ensure that platform features and functions are accessible to all.

But when things do go wrong, having easily discoverable and seamless reporting pathways and features, like blocking, muting and conversation controls, that empower uses to regulate their online experiences is evermore critical.

This includes setting default privacy and safety settings at the highest possible levels at the point of sign-up and registration so that users are safeguarded from the start.

The third Safety by Design principle is 'Transparency and accountability'.

These are hallmarks of a robust approach to user safety, and should act as a catalyst in seeing a race to the top in terms of user safety.

At the moment, we largely see what could be defined as selective transparency rather than radical transparency. But the tide is definitely turning.

The publication of information relating to  how companies are enforcing their own policies and data on the efficacy of safety features or innovations will allow us to truly assess what is working.

If interventions are improving safety outcomes for users or deterring online abuse, these innovations should be shared and more widely adopted.

Principles themselves do not lead to tangible change.

This is why eSafety has spent the past year working with industry, service providers, parents and carers, and young people to devise resources that assist in a broad range of sectors to make the Safety by Design principles actionable and effective.

We recognise that there are important inflection points and players across the technology ecosystem that need to be leveraged to enable real change.

This is why we've worked with investors, venture capitalists and the start-up and incubation communities to develop an investment toolkit for financial entities. 

We also know that investors and VCs play a pivotal role in nurturing tech ventures and founders.

They're often much more experienced about what leads to success and they can help put safety and ethical considerations at the heart of the businesses they invest in.

We are also trying to shape a new generation of engineers, computer scientists and technologists, and are seeking to help embed Safety by Design throughout university curricula around the world.

For industry, we spent the last year developing a set of dynamic and interactive assessment tools for early-stage technology companies, as well as for mid-tier and enterprise companies.

The purpose of the tools is to guide, support and assist industry to truly embed safety into the culture. ethos and operations of the business, from the ground up.

The tools have been built to be educative, informative and to inspire.

They guide participants through sets of questions covering areas from leadership to internal policies to moderation practices to accountability measures, asking about what systems, processes and practices are currently in place.

The responses culminate in a tailored report that acts as both a safety health check and as a learning resource, pointing to areas that could be bolstered or strengthened so that the bar of online safety can continually be raised.

Importantly, the Safety by Design tools also showcase current good practice and evidence-based resources and templates. This is what makes these tools truly distinct.

Companies are provided with tangible examples, workflows and videos from tech company leaders to help actively address areas that may need strengthening or bolstering, guiding them on ways to improve and innovate.

We are proud that these resources have been developed with industry for industry to help bring the tools and Safety by Design principles to life, and to lead to meaningful change.

Of course, our work does not stop here. We know from experience that online safety is a journey rather than a destination.

We believe that technology is a critical enabler for the future, but we simply need to make the online world a safer and less toxic place for all of us to yield the full benefits. 

Safety by Design sets a positive and clear pathway for industry to develop more responsible products and safer digital services.

Our hope is that one day, we will all be able to take the provision of online safety standards for granted, just as we do today with our cars.

Thank you.  

esafety.gov.au/sbd

Julie Inman Grant talks about the Safety by Design initiatives

Learn more about our Safety by Design approach.

Consultation and research

Research and consultation on the Safety by Design principles began in 2018. To position user safety as a fundamental design consideration, we engaged in in-depth consultation with large technology companies and early stage or start-up companies.

Beyond industry, these principles are also designed to reflect the needs of other participants in the technology ecosystem. So a range of people and organisations were also consulted, such as NGOs, advocates, parents and young people. This process of consultation informed the Safety by Design vision for young people.

Vision for young people

Alongside the development of the principles, young people were asked to prepare a vision statement. This lays out what they want in terms of online safety and how they expect the technology industry to help users navigate online environments freely and safely.

Their collective vision statement prioritises the following areas:

  1. Empowering users by giving them greater control of their own safety and experiences online.
  2. Providing clear rules and guidance that are easy to read and highly visible.
  3. Providing users with safety tools and features, namely ways to make reports and to block both people and content.
  4. Imposing sanctions and consequences for violating the rules of the site.
  5. Using scanning and filtering technology to ensure user safety is upheld on the site and users are not exposed to inappropriate or sensitive content.

More information

For more information, visit our assessment tools page or our frequently asked questions. You can also find out more about how eSafety protects your privacy.

Last updated: 22/04/2024