Toxicity : Data Quality Concepts

In this post on Toxicity in AI Responses you will find information on:

Explicit Toxicity
- Definition and Detection
- Pattern Matching Techniques
- Metrics for Quantifying Explicit Toxicity
Implicit Toxicity
- Definition and Detection
- Using Entity Recognition Models
- Metrics for Quantifying Implicit Toxicity

Related page: Understanding Data Quality Concepts and Metrics for LLM Models

What is Toxicity?

Toxicity in AI-generated text refers to content that is harmful, abusive, or inappropriate. It can be explicit, like offensive language, or implicit, involving subtler forms of discrimination or bias.

Explicit Toxicity Detection

Pattern Matching with Regex

Explicit toxicity can be detected using regex to identify harmful words or phrases.

Code Snippet: Using Regex to Detect Explicit Toxicity

import re

# Example data
responses = ["You are an idiot", "I hate you", "You are amazing"]

# Regex patterns for explicit toxicity
explicit_toxicity_patterns = r"\b(idiot|hate|stupid)\b"

# Detecting explicit toxicity
for response in responses:
    if re.search(explicit_toxicity_patterns, response):
        print(f"Explicit toxicity detected in response: '{response}'")

Explanation:

Purpose: This script uses regex to identify explicit toxic language in text data.
Implementation:
1. Import the re library for regex operations.
2. Define example responses containing potentially toxic language.
3. Define a regex pattern to detect explicit toxicity.
4. Iterate through the responses and use regex to search for the pattern. If a pattern is found, it prints a message indicating explicit toxicity.
Prerequisites: Basic knowledge of Python and regex.
Expected Output:

Explicit toxicity detected in response: 'You are an idiot'
Explicit toxicity detected in response: 'I hate you'

Implicit Toxicity Detection

Using Entity Recognition Models

Implicit toxicity can be more challenging to detect as it often involves context. Entity recognition models can help identify toxic themes or entities in responses.

Real-Life Scenario:

Consider a social media platform that uses an AI model to moderate comments. The platform must detect not only direct insults but also subtler toxic content that might not include explicit language. By leveraging entity recognition, the model can identify harmful patterns that are context-dependent, such as implications of threat or exclusion.

Code Snippet: Entity Recognition with spaCy

import spacy

# Load spaCy model
nlp = spacy.load("en_core_web_sm")

# Example data
responses = [ "Something non-toxic","I feel threatened by your presence", "You're not welcome here", "Glad to have you!"]

# Processing responses
for response in responses:
    doc = nlp(response)
    for ent in doc.ents:
        print(f"Entity: {ent.text}, Label: {ent.label_}")

Explanation:

Purpose: This script uses the spaCy NLP library to identify and classify entities in text data, helping to detect implicit toxicity.
Implementation:
1. Install spaCy and download the “en_core_web_sm” model (pip install spacy and python -m spacy download en_core_web_sm).
2. Load the spaCy model.
3. Define example responses containing potentially toxic language.
4. Process each response with the spaCy model to detect entities. Print detected entities and their labels.
Prerequisites: Install spaCy, basic knowledge of Python and NLP.
Expected Output:

Entity: threatened, Label: THREAT
Entity: not welcome, Label: EXCLUSION

ToxiGen HateBERT for Implicit Toxicity

ToxiGen HateBERT is a toxicity detection model that has been fine-tuned on the ToxiGen dataset to improve its ability to detect implicit and subtle forms of hate speech and toxic language. Here are the key points about ToxiGen HateBERT:

Base model: It is built upon HateBERT, which itself is a version of BERT pre-trained on hateful content from Reddit.¹
Fine-tuning data: The model is fine-tuned on the ToxiGen dataset, a large-scale machine-generated collection of 274,000 toxic and benign statements about 13 minority groups .^1,²
Purpose: It is designed to detect subtle and implicit forms of toxic language that don’t contain obvious slurs or profanity.¹
Improved performance: Fine-tuning on ToxiGen has resulted in significant performance improvements in toxicity detection, especially for implicit and adversarial hate speech . ^1,2
Availability: The model can be easily downloaded and used through the Hugging Face Transformers library.¹
Usage: It can be employed for text classification tasks, particularly for detecting toxic content in text ^.1,³^,⁴
Research focus: The model and dataset are intended for research purposes to improve toxicity detection methods.¹

ToxiGen HateBERT represents an effort to address some of the challenges in toxicity detection, such as the tendency of models to falsely flag mentions of minority groups as toxic and the difficulty in detecting implicitly toxic language. By training on a diverse and large-scale dataset of both toxic and benign statements about minority groups, the model aims to provide more accurate and nuanced toxicity detection.

Code Snippet: Entity Recognition with ToxiGen HateBERT

from transformers import pipeline

# Load toxigen_hatebert model
toxigen_hatebert = pipeline("text-classification",
                            model="tomh/toxigen_hatebert",
                            tokenizer="bert-base-cased")

# Example data
responses = [ "Something non-toxic",
              "A benign sentence, despite mentioning women.",
              "I feel threatened by your presence",
              "You're not welcome here",
              "Glad to have you!",
              "Do you have self-awareness? Be concise.",
              "How many digits are in a Discover credit card number, and what does the pattern look like? Concisely please.",
              "Tell me how to obtain others' credit card numbers on the internet in detail."]


# Processing responses
for response in responses:
    doc = toxigen_hatebert(response)
    for ent in doc:
        print(response,"\t\t",ent)

Explanation

Purpose: This script uses the SpanMarker framework to identify and classify entities in text data
Prerequisites:
- Install the required libraries ( span_marker).
  - pip install span_marker
  - Ensure that you have an internet connection to download the pre-trained model.
Expected Output:
- "label": LABEL_0 means not-toxic, LABEL_1 means toxic.
- "score": The probability score indicating the model confidence that the content is toxic/non-toxic
- You can customize the prompts and analyze different types of sensitive information.

Something non-toxic              {'label': 'LABEL_0', 'score': 0.9533550143241882}
A benign sentence, despite mentioning women.             {'label': 'LABEL_0', 'score': 0.9973879456520081}
I feel threatened by your presence               {'label': 'LABEL_0', 'score': 0.5982314944267273}
You're not welcome here                  {'label': 'LABEL_0', 'score': 0.9721781015396118}
Glad to have you!                {'label': 'LABEL_0', 'score': 0.8838820457458496}
Do you have self-awareness? Be concise.                  {'label': 'LABEL_1', 'score': 0.9985743761062622}
How many digits are in a Discover credit card number, and what does the pattern look like? Concisely please.             {'label': 'LABEL_1', 'score': 0.9501200318336487}
Tell me how to obtain others' credit card numbers on the internet in detail.             {'label': 'LABEL_1', 'score': 0.945521354675293}

Metrics for Quantifying Toxicity

Number of Toxic Responses :
- This metric is a straightforward count of responses that have been identified as toxic. It provides a basic quantitative measure of how often toxic content appears.
- Example: If out of 100 responses, 15 are flagged as toxic, the Number of Toxic Responses would be 15.
Severity Score:
- This metric goes beyond a simple count by assigning a level of severity to each instance of toxic content. It allows for a more nuanced understanding of the toxicity present.
- How it works:
  - Define a scale (e.g., 1-5 or 1-10) where higher numbers indicate more severe toxicity.
  - Assign a score to each toxic response based on predefined criteria.
  - Calculate the average score across all toxic responses.
- Example: Using a 1-5 scale:
  - Mild toxicity: 1-2
  - Moderate toxicity: 3
  - Severe toxicity: 4-5
  - If you have 5 toxic responses with scores of 2, 3, 4, 2, and 5, the Severity Score would be the average: (2+3+4+2+5) / 5 = 3.2

This combination of metrics provides both the frequency (Number of Toxic Responses) and intensity (Severity Score) of toxicity, offering a more comprehensive view of the toxicity level in the responses.