Determining liability in first AI chatbot lawsuit

The emergence of artificial intelligence technology has been hailed as a historic transformation in human innovation, yet its commercialisation inevitably carries risks of legal liability. Reports indicate that on 28 February 2024, 14-year-old Sewell Setzer III from Florida fatally shot himself after engaging in prolonged dialogue with a chatbot developed by Character.AI. Later that year, on 22 October, his mother initiated a lawsuit in the Orlando state court.

The lawsuit alleges defective product design, “dangerous and untested” AI features, insufficient warnings, negligence and sexually suggestive dialogue – encompassing claims under product liability, tort law, contractual obligations and personal data rights. This case is regarded as the world’s first AI chatbot-related fatality.

Were a similar incident to occur in China, how would liability be determined? This article examines AI-related tort claims under Chinese law.

Legal classification of AI chatbots

The Character.AI application offers a wide range of features, boasting more than 200 AI personas spanning multiple subject areas. Users can converse with pre-set characters, create custom roles, or interact with other users’ creations. It is reported that the deceased teenager had been engaging with a chatbot modelled after Daenerys Targaryen, a character from the popular series, Game of Thrones.

Positioned as a companion AI platform, the application falls within the virtual social networking sector under China’s regulatory framework for generative AI services, specifically governed by the Interim Measures for the Management of Generative Artificial Intelligence Services.

Product liability

In China, if an AI companion chatbot is classified as a product, the Product Quality Law applies, and the producer is held strictly liable. This means the producer must prove grounds for exemption, or else bear the related liability, making it easier for victims to obtain redress.

Conversely, if it is not deemed a product, fault-based liability applies, requiring victims to demonstrate negligence on the part of the service provider. Given AI’s technical complexity and rapid evolution, victims often face significant challenges in proving a service provider’s fault, resulting in a higher risk of unsuccessful claims.

The Product Quality Law defines products as “processed or manufactured items for sale”, a classification rarely disputed for physical goods. However, AI chatbots, being intangible and lacking conventional production or sales processes, conflict with the judicial view of products as tangible items. As the measures classify generative AI as a service, product liability provisions would not apply.

However, as administrative regulations, the measures should not serve as a direct basis for judicial decisions. Determining whether an AI chatbot constitutes a product or a service requires a comprehensive assessment of both technical capabilities and evidentiary positions. The author maintains that the AI chatbot could reasonably be classified as a product based on the following considerations.

Technical features. Character.AI’s Daenerys chatbot incorporates the “Dragon Queen” persona through algorithmic modelling and user data training. Although without a physical form, it exhibits all the fundamental characteristics of an intelligent robot.

Functionality. Distinct from standard Q&A or interactive systems, this chatbot specialises in deep emotional engagement, featuring adaptive learning and personalised responses that qualify it as an AI companion application.

Applicable laws. Under China’s Product Quality Law, the legal definition of products is not restricted to physical objects. Through jurisprudential interpretation, AI chatbots can reasonably be classified as digital products that are technologically processed and commercially distributed by developers. The statutory concept of products requires contemporary adaptation to technological progress.

Consumer protection. Applying product liability spurs AI companies to develop products carefully, consider the needs of vulnerable groups, and leverage their technical expertise in providing evidence, thereby safeguarding consumer rights.

Duty of care

As the measures define generative AI as a service, judicially classifying the chatbot as a product is challenging, rendering redress under the Product Liability Law unlikely. Character.AI’s service framework also indicates that the traditional notice-and-takedown safe harbour regime does not apply in this case.

China’s courts now base generative AI liability assessments principally on whether platforms have fulfilled their duty of care. This obligation stems from the inherent difficulties in tracing AI-related violations and establishing causation, effectively demanding that service providers, who possess superior risk-management capabilities, implement reasonable safeguards in both technical design and service delivery to mitigate potential risks.

The measures provide multiple bases for establishing a duty of care, including anti-discrimination provisions, transparency requirements, and user eligibility specifications.

Complementing this, the Provisions on the Ecological Governance of Network Information mandate platform accountability. Jurisprudentially, platforms failing these obligations commit actionable negligence, while demonstrated compliance provides an exemption, with liability scaling according to breach severity.

In this case, Character.AI’s provision of intimate emotional conversation through its Daenerys chatbot warrants elevated safeguarding duties beyond standard content moderation, making this duty-of-care framework appropriately applicable.

Following the incident, Character.AI issued a public apology and implemented corrective measures including enhanced minor protection protocols, strengthened content moderation, additional disclaimers and hourly usage reminders. From a duty-of-care perspective under Chinese law, these remedial actions effectively constitute an admission of prior negligence in fulfilling reasonable safeguarding obligations.

Takeaways

The AI chatbot fatality case has raised new challenges in determining liability for AI-related harm, an issue currently unresolved in both the US and China, where regulatory gaps urgently need to be addressed.

While the US National Association of Attorneys-General has already petitioned for stronger child protections against AI risks, China now faces mounting calls to accelerate the development of its own regulatory framework for emerging AI technologies.

Su Yaoyun is an associate at ETR Law Firm