Organize documentation: Move all .md files to readmes/ directory
This commit is contained in:
315
readmes/TESTING_V2.md
Normal file
315
readmes/TESTING_V2.md
Normal file
@@ -0,0 +1,315 @@
|
||||
# Testing Autonomous System V2
|
||||
|
||||
## Quick Start Guide
|
||||
|
||||
### Step 1: Enable V2 System (Optional - Test Mode)
|
||||
|
||||
The V2 system can run **alongside** V1 for comparison. To enable it:
|
||||
|
||||
**Option A: Edit `bot.py` to start V2 on bot ready**
|
||||
|
||||
Add this to the `on_ready()` function in `bot/bot.py`:
|
||||
|
||||
```python
|
||||
# After existing setup code, add:
|
||||
from utils.autonomous_v2_integration import start_v2_system_for_all_servers
|
||||
|
||||
# Start V2 autonomous system
|
||||
await start_v2_system_for_all_servers(client)
|
||||
```
|
||||
|
||||
**Option B: Manual API testing (no code changes needed)**
|
||||
|
||||
Just use the API endpoints to check what V2 is thinking, without actually running it.
|
||||
|
||||
### Step 2: Test the V2 Decision System
|
||||
|
||||
#### Check what V2 is "thinking" for a server:
|
||||
|
||||
```bash
|
||||
# Get current social stats
|
||||
curl http://localhost:3939/autonomous/v2/stats/<GUILD_ID>
|
||||
|
||||
# Example response:
|
||||
{
|
||||
"status": "ok",
|
||||
"guild_id": 759889672804630530,
|
||||
"stats": {
|
||||
"loneliness": "0.42",
|
||||
"boredom": "0.65",
|
||||
"excitement": "0.15",
|
||||
"curiosity": "0.20",
|
||||
"chattiness": "0.70",
|
||||
"action_urgency": "0.48"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Trigger a manual V2 analysis:
|
||||
|
||||
```bash
|
||||
# See what V2 would decide right now
|
||||
curl http://localhost:3939/autonomous/v2/check/<GUILD_ID>
|
||||
|
||||
# Example response:
|
||||
{
|
||||
"status": "ok",
|
||||
"guild_id": 759889672804630530,
|
||||
"analysis": {
|
||||
"stats": { ... },
|
||||
"interest_score": "0.73",
|
||||
"triggers": [
|
||||
"KEYWORD_DETECTED (0.60): Interesting keywords: vocaloid, miku",
|
||||
"CONVERSATION_PEAK (0.60): Lots of people are chatting"
|
||||
],
|
||||
"recent_messages": 15,
|
||||
"conversation_active": true,
|
||||
"would_call_llm": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Get overall V2 status:
|
||||
|
||||
```bash
|
||||
# See V2 status for all servers
|
||||
curl http://localhost:3939/autonomous/v2/status
|
||||
|
||||
# Example response:
|
||||
{
|
||||
"status": "ok",
|
||||
"servers": {
|
||||
"759889672804630530": {
|
||||
"server_name": "Example Server",
|
||||
"loop_running": true,
|
||||
"action_urgency": "0.52",
|
||||
"loneliness": "0.30",
|
||||
"boredom": "0.45",
|
||||
"excitement": "0.20",
|
||||
"chattiness": "0.70"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Monitor Behavior
|
||||
|
||||
#### Watch for V2 log messages:
|
||||
|
||||
```bash
|
||||
docker compose logs -f bot | grep -E "🧠|🎯|🤔"
|
||||
```
|
||||
|
||||
You'll see messages like:
|
||||
```
|
||||
🧠 Starting autonomous decision loop for server 759889672804630530
|
||||
🎯 Interest score 0.73 - Consulting LLM for server 759889672804630530
|
||||
🤔 LLM decision: YES, someone mentioned you (Interest: 0.73)
|
||||
```
|
||||
|
||||
#### Compare V1 vs V2:
|
||||
|
||||
**V1 logs:**
|
||||
```
|
||||
💬 Miku said something general in #miku-chat
|
||||
```
|
||||
|
||||
**V2 logs:**
|
||||
```
|
||||
🎯 Interest score 0.82 - Consulting LLM
|
||||
🤔 LLM decision: YES
|
||||
💬 Miku said something general in #miku-chat
|
||||
```
|
||||
|
||||
### Step 4: Tune the System
|
||||
|
||||
Edit `bot/utils/autonomous_v2.py` to adjust behavior:
|
||||
|
||||
```python
|
||||
# How sensitive is the decision system?
|
||||
self.LLM_CALL_THRESHOLD = 0.6 # Lower = more responsive (more LLM calls)
|
||||
self.ACTION_THRESHOLD = 0.5 # Lower = more chatty
|
||||
|
||||
# How fast do stats build?
|
||||
LONELINESS_BUILD_RATE = 0.01 # Higher = gets lonely faster
|
||||
BOREDOM_BUILD_RATE = 0.01 # Higher = gets bored faster
|
||||
|
||||
# Check intervals
|
||||
MIN_SLEEP = 30 # Seconds between checks during active chat
|
||||
MAX_SLEEP = 180 # Seconds between checks when quiet
|
||||
```
|
||||
|
||||
### Step 5: Understanding the Stats
|
||||
|
||||
#### Loneliness (0.0 - 1.0)
|
||||
- **Increases**: When not mentioned for >30 minutes
|
||||
- **Decreases**: When mentioned, engaged
|
||||
- **Effect**: At 0.7+, seeks attention
|
||||
|
||||
#### Boredom (0.0 - 1.0)
|
||||
- **Increases**: When quiet, hasn't spoken in >1 hour
|
||||
- **Decreases**: When shares content, conversation happens
|
||||
- **Effect**: At 0.7+, likely to share tweets/content
|
||||
|
||||
#### Excitement (0.0 - 1.0)
|
||||
- **Increases**: During active conversations
|
||||
- **Decreases**: Fades over time (decays fast)
|
||||
- **Effect**: Higher = more likely to jump into conversation
|
||||
|
||||
#### Curiosity (0.0 - 1.0)
|
||||
- **Increases**: Interesting keywords detected
|
||||
- **Decreases**: Fades over time
|
||||
- **Effect**: High curiosity = asks questions
|
||||
|
||||
#### Chattiness (0.0 - 1.0)
|
||||
- **Set by mood**:
|
||||
- excited/bubbly: 0.85-0.9
|
||||
- neutral: 0.5
|
||||
- shy/sleepy: 0.2-0.3
|
||||
- asleep: 0.0
|
||||
- **Effect**: Base multiplier for all interactions
|
||||
|
||||
### Step 6: Trigger Examples
|
||||
|
||||
Test specific triggers by creating conditions:
|
||||
|
||||
#### Test MENTIONED trigger:
|
||||
1. Mention @Miku in the autonomous channel
|
||||
2. Check stats: `curl http://localhost:3939/autonomous/v2/check/<GUILD_ID>`
|
||||
3. Should show: `"triggers": ["MENTIONED (0.90): Someone mentioned me!"]`
|
||||
|
||||
#### Test KEYWORD trigger:
|
||||
1. Say "I love Vocaloid music" in channel
|
||||
2. Check stats
|
||||
3. Should show: `"triggers": ["KEYWORD_DETECTED (0.60): Interesting keywords: vocaloid, music"]`
|
||||
|
||||
#### Test CONVERSATION_PEAK:
|
||||
1. Have 3+ people chat within 5 minutes
|
||||
2. Check stats
|
||||
3. Should show: `"triggers": ["CONVERSATION_PEAK (0.60): Lots of people are chatting"]`
|
||||
|
||||
#### Test LONELINESS:
|
||||
1. Don't mention Miku for 30+ minutes
|
||||
2. Check stats: `curl http://localhost:3939/autonomous/v2/stats/<GUILD_ID>`
|
||||
3. Watch loneliness increase over time
|
||||
|
||||
### Step 7: Debugging
|
||||
|
||||
#### V2 won't start?
|
||||
```bash
|
||||
# Check if import works
|
||||
docker compose exec bot python -c "from utils.autonomous_v2 import autonomous_system_v2; print('OK')"
|
||||
```
|
||||
|
||||
#### V2 never calls LLM?
|
||||
```bash
|
||||
# Check interest scores
|
||||
curl http://localhost:3939/autonomous/v2/check/<GUILD_ID>
|
||||
|
||||
# If interest_score is always < 0.6:
|
||||
# - Channel might be too quiet
|
||||
# - Stats might not be building
|
||||
# - Try mentioning Miku (instant 0.9 score)
|
||||
```
|
||||
|
||||
#### V2 calls LLM too much?
|
||||
```bash
|
||||
# Increase threshold in autonomous_v2.py:
|
||||
self.LLM_CALL_THRESHOLD = 0.7 # Was 0.6
|
||||
```
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### Expected LLM Call Frequency
|
||||
|
||||
**Quiet server (few messages):**
|
||||
- V1: ~10 random calls/day
|
||||
- V2: ~2-5 targeted calls/day
|
||||
- **GPU usage: LOWER with V2**
|
||||
|
||||
**Active server (100+ messages/day):**
|
||||
- V1: ~10 random calls/day (same)
|
||||
- V2: ~10-20 targeted calls/day (responsive to activity)
|
||||
- **GPU usage: SLIGHTLY HIGHER, but much more relevant**
|
||||
|
||||
### Check GPU Usage
|
||||
|
||||
```bash
|
||||
# Monitor GPU while bot is running
|
||||
nvidia-smi -l 1
|
||||
|
||||
# V1: GPU spikes randomly every 15 minutes
|
||||
# V2: GPU spikes only when something interesting happens
|
||||
```
|
||||
|
||||
### Monitor LLM Queue
|
||||
|
||||
If you notice lag:
|
||||
1. Check how many LLM calls are queued
|
||||
2. Increase `LLM_CALL_THRESHOLD` to reduce frequency
|
||||
3. Increase check intervals for quieter periods
|
||||
|
||||
## Migration Path
|
||||
|
||||
### Phase 1: Testing (Current)
|
||||
- V1 running (scheduled actions)
|
||||
- V2 running (parallel, logging decisions)
|
||||
- Compare behaviors
|
||||
- Tune V2 parameters
|
||||
|
||||
### Phase 2: Gradual Replacement
|
||||
```python
|
||||
# In server_manager.py, comment out V1 jobs:
|
||||
# scheduler.add_job(
|
||||
# self._run_autonomous_for_server,
|
||||
# IntervalTrigger(minutes=15),
|
||||
# ...
|
||||
# )
|
||||
|
||||
# Keep V2 running
|
||||
autonomous_system_v2.start_loop_for_server(guild_id, client)
|
||||
```
|
||||
|
||||
### Phase 3: Full Migration
|
||||
- Disable all V1 autonomous jobs
|
||||
- Keep only V2 system
|
||||
- Keep manual triggers for testing
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Module not found: autonomous_v2"
|
||||
```bash
|
||||
# Restart the bot container
|
||||
docker compose restart bot
|
||||
```
|
||||
|
||||
### "Stats always show 0.00"
|
||||
- V2 decision loop might not be running
|
||||
- Check: `curl http://localhost:3939/autonomous/v2/status`
|
||||
- Should show: `"loop_running": true`
|
||||
|
||||
### "Interest score always low"
|
||||
- Channel might be genuinely quiet
|
||||
- Try creating activity: post messages, images, mention Miku
|
||||
- Loneliness/boredom build over time (30-60 min)
|
||||
|
||||
### "LLM called too frequently"
|
||||
- Increase thresholds in `autonomous_v2.py`
|
||||
- Check which triggers are firing: use `/autonomous/v2/check`
|
||||
- Adjust trigger scores if needed
|
||||
|
||||
## API Endpoints Reference
|
||||
|
||||
```
|
||||
GET /autonomous/v2/stats/{guild_id} - Get social stats
|
||||
GET /autonomous/v2/check/{guild_id} - Manual analysis (what would V2 do?)
|
||||
GET /autonomous/v2/status - V2 status for all servers
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Run V2 for 24-48 hours
|
||||
2. Compare decision quality vs V1
|
||||
3. Tune thresholds based on server activity
|
||||
4. Gradually phase out V1 if V2 works well
|
||||
5. Add dashboard for real-time stats visualization
|
||||
Reference in New Issue
Block a user