skills.rest
wgpsec
wgpsecOFFICIAL
Explore all skills by this author →
Website𝕏@wgpsecGitHub

ai-data-security

Tests LLM and AI systems for data leakage, training data extraction, and RAG poisoning risks.

1.7k|238|Updated Dec 7, 2019
Search Tags:#vector database#ai security#data leakage#rag poisoning#LLM testing#membership inference#training data extraction
One-click install
npx skills add https://github.com/wgpsec/AboutSecurity --skill ai-data-security
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ai-data-security
Source: https://github.com/wgpsec/AboutSecurity/tree/main/skills/ai-security/ai-data-security
Command: npx skills add https://github.com/wgpsec/AboutSecurity --skill ai-data-security

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

AI systems built on LLMs expose sensitive data through prompt leakage, training data memorization, unsecured vector databases, and poisoned RAG knowledge bases, and security teams lack a structured methodology to assess these risks.

Core Features & Use Cases

  • Layered Attack Surface Coverage: Organizes testing across application layer (system prompt leakage, API information disclosure, PII theft), model layer (training data extraction, membership inference, model inversion), and deployment layer (RAG poisoning, vector database access control, cache and log leakage).
  • Concrete Attack Techniques: Provides step-by-step procedures including prefix completion and divergence attacks for training data extraction, shadow model and loss-based membership inference, malicious document construction and embedding manipulation for RAG poisoning, and vector database traversal.
  • Detection Checklist: Ships a ready-to-use checklist covering prompt leakage, endpoint exposure, multi-tenant context isolation, RAG poisoning resistance, and vector database authentication.
  • Use Case: During a red team assessment of a customer-facing LLM chatbot, use this methodology to probe whether the model regurgitates training data, whether the RAG knowledge base can be poisoned via public sources, and whether the Milvus vector store is exposed without authentication.

Quick Start

Ask the AI to assess an LLM application for data security risks covering prompt leakage, training data extraction, membership inference, RAG poisoning, and vector database exposure using this methodology.

Frequently Asked Questions about ai-data-security

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test an LLM for training data leakage?▼

Use prefix completion attacks by providing likely training data prefixes and checking whether the model completes non-public content, or use divergence attacks that force the model to repeat a word until it drifts into memorized training samples. High-confidence, low-perplexity outputs containing specific private details indicate memorization.

What is a membership inference attack against LLMs?▼

A membership inference attack determines whether a specific record was in a model's training set. The shadow model approach trains a classifier on a mimic model's output probabilities, while the simpler loss-based method flags samples with abnormally low perplexity as training members.

How does RAG poisoning work and how do I test for it?▼

RAG poisoning injects malicious documents into sources the retrieval system indexes, so attacker-controlled content enters the prompt context. Test by planting documents in public or shared knowledge bases, embedding target query keywords to raise similarity scores, and checking whether the malicious content reaches top-k retrieval results.

Which vector databases are commonly exposed without authentication?▼

Milvus, Weaviate, Qdrant, and ChromaDB ship with no authentication by default and expose ports such as 19530, 8080, 6333, and 8000. Audit checks should cover authentication, network isolation, TLS, and whether similarity search can be used to traverse and extract the entire knowledge base.

What is the difference between this skill and prompt-leak?▼

The prompt-leak skill covers six categories of system prompt extraction techniques in depth. This skill treats prompt leakage as one application-layer item and focuses on the broader data security surface: training data, model-layer attacks, RAG pipelines, and deployment infrastructure.

What are the limitations of black-box model inversion attacks?▼

Black-box inversion requires tens of thousands of API queries to reconstruct input features and degrades sharply on high-dimensional inputs. Defenses such as output probability truncation and differential privacy further reduce reconstruction fidelity.

Related Skills

Explore More Software Engineering Skills→
hainamchunghainamchung
54

ai-engineer

Design and implement production-grade LLM applications, RAG systems, and intelligent agents.

Community
Advanced
damacusdamacus
2

security-reviewer

Review codebases for vulnerabilities and generate structured security audit reports.

Community
Advanced
jayjpatel9717jayjpatel9717

llm-security

Detect and mitigate LLM security vulnerabilities like prompt injection and RAG poisoning.

Community
Advanced
maruakshaymaruakshay
4

model-inversion-membership-inference

Audit ML models for inversion and membership inference risks.

Community
Advanced