스킬 에이전트의 KV 캐시 재사용: 지금까지 한 실험

스킬 에이전트는 요청마다 긴 스킬 글을 프롬프트에 넣습니다. 이 발표는 그 글의 계산 결과를 얼마나 다시 쓸 수 있는지 잰 실험을 보고합니다. 마지막에는 연구 질문에 대한 고민과 3년 계획 주제를 나눕니다.

이 글에 대해

2026-11-06 연구실 세미나에서 발표한 자료입니다. 실험은 H100 80GB 1장(VESSL)과 vLLM 0.11.0에서 했습니다. 1장은 기본 지식, 2장은 실험, 3장과 4장은 연구 질문에 대한 고민과 3년 계획입니다. 수치는 진행 중인 실험의 결과입니다.

1-1 용어: 실제 프롬프트 하나로 설명합니다

실험 환경은 ALFWorld입니다. ALFWorld는 글로 하는 집안일 게임입니다. 모델은 방 설명을 읽고 go to sinkbasin 1 같은 명령을 보냅니다. 아래는 실제 게임 한 판의 처음 세 번입니다.

에피소드

게임 한 판입니다. 과제에 성공하거나 행동을 50번 하면 끝납니다.

스텝

에피소드 안의 행동 한 번입니다. 스텝마다 모델에 요청을 한 번 보냅니다. 스텝 번호는 0부터 셉니다.

카테고리

과제의 종류입니다. SkillRL은 과제 문장의 낱말을 보고 6종 중 하나를 고릅니다. 예를 들어 "clean"이 있으면 clean 카테고리입니다. 에피소드 안에서는 바뀌지 않습니다.

스킬

프롬프트에 넣는 행동 요령 글입니다. 예: "Sink First for Cleaning: 물건을 들면 바로 싱크대로 가서 씻는다."

스킬 뱅크

스킬을 모아 둔 JSON 파일입니다. SkillRL의 ALFWorld 뱅크에는 스킬 55개가 있습니다.

프롬프트 구간

프롬프트 안에서 하는 일이 다른 부분입니다. 아래 그림에서 색이 다른 부분이 각 구간입니다.

run

게임 64개를 동시에 끝까지 돌린 실험 한 번입니다.

seed

모델이 응답을 고를 때 쓰는 난수의 시작값입니다. 모델은 다음 낱말을 확률에 따라 뽑습니다 (temperature 0.4). 그래서 seed가 다르면 같은 게임에서도 다른 응답과 행동이 나옵니다. 게임 64개는 seed와 관계없이 같습니다. 같은 조건을 seed 0·1·2로 세 번 돌리고 평균과 표준편차를 봅니다.

아래는 위 에피소드의 step 2 요청 전문입니다. 색은 구간의 종류를 나타냅니다. 숫자는 토큰 수입니다.

1. Qwen 기본 system 문장 16
2. Role sentence 16
3. task 문장 11
4. General Principles 218
5. 카테고리 스킬 (Clean Skills) 375
6. Mistakes to Avoid 278
7. Progress header (step count) 34
8. 이력 첫 항목 148
9. 이력 둘째 항목 54
10. 현재 관찰 17
11. 가능한 행동 목록 144
12. 지시문 67

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: put a clean cloth in cabinet.

## Retrieved Relevant Experience

### General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.

### Clean Skills
- **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
  _Apply when: As soon as the goal specifies the object must be clean before placement._
- **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
  _Apply when: After visually confirming the presence of the target object on any surface or inside an opened container._
- **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
  _Apply when: Once the target object is in hand and its required state is ‘clean’._
- **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
  _Apply when: After initial obvious surfaces are empty and the target object remains unfound._
- **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
  _Apply when: Immediately after a cleaning action and before placing the object at its target location._
- **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.
  _Apply when: At the very start of the task to choose the first search destination for the target object._

### Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.

## Current Progress

Prior to this step, you have already taken 2 step(s). Below are the most recent 2 observations and the corresponding actions you took: [Observation 1: '-= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.', Action 1: 'go to handtowelholder 1']
[Observation 2: 'You arrive at handtowelholder 1. On the handtowelholder 1, you see a handtowel 1.', Action 2: 'take handtowel 1 from handtowelholder 1']
You are now at step 3 and your current observation is: You pick up the handtowel 1 from the handtowelholder 1.
Your admissible actions of the current situation are: ['examine handtowel 1'
 'examine handtowelholder 1'
 'go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'move handtowel 1 to handtowelholder 1'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant

이 요청은 1,421토큰입니다. 그중 스킬이 871토큰입니다. 매 스텝 이 프롬프트 전체를 새로 만들어 보냅니다.

KV 캐시

모델이 프롬프트를 읽으며 토큰마다 만드는 중간 계산 결과입니다. 응답을 만들 때 이 값을 씁니다.

재사용

앞 요청에서 만든 KV 캐시를 다음 요청에서 다시 계산하지 않고 가져다 쓰는 일입니다.

재사용률

캐시에서 가져온 프롬프트 토큰 수를 전체 프롬프트 토큰 수로 나눈 값입니다. 1에 가까울수록 계산을 적게 합니다.

다시 계산한 토큰

캐시에 없어서 모델이 새로 계산한 프롬프트 토큰 수입니다.

모델은 세 개를 씁니다. 학습하지 않은 Qwen2.5-7B-Instruct(원본 모델), SkillRL의 SFT 모델, SkillRL의 RL 모델입니다. RL 모델이 SkillRL 논문의 최종 모델입니다.

1-2 세 논문: 무엇을 풀었고 어떻게 풀었나

세 시스템은 모두 경험에서 스킬을 만들고, 그 스킬을 프롬프트에 넣습니다. 가장 큰 차이는 무엇을 바꾸는가입니다. 아래 그림에서 주황 상자가 각 시스템이 바꾸는 쪽입니다.

출처: docs/study/skill-systems-compare.html 그림 1과 7절 비교표.

SkillRL (arXiv 2602.08234, NeurIPS 2026)

푼 문제

기존 방법은 경험을 궤적이나 메모 그대로 프롬프트에 넣습니다 (ExpeL, Mem0 등). SkillRL은 궤적을 짧은 스킬로 정리합니다. 그리고 모델이 그 스킬을 잘 쓰도록 강화학습을 합니다 PAPERS.md.

학습 과정

아래 흐름의 ①~④를 거칩니다. ⑤는 학습 중에만 일어납니다.

스킬을 넣는 방법

에피소드 시작 때 과제 문장의 낱말로 카테고리를 고릅니다. 그다음 일반 스킬 앞 6개, 그 카테고리의 스킬 전부, 실수 앞 5개를 user 메시지에 넣습니다. step 0 요청에는 스킬을 넣지 않습니다.

프롬프트 구조

매 스텝 user 메시지 하나를 처음부터 새로 만듭니다. 이전 응답은 넣지 않고, 최근 이력 2개만 넣습니다.

성능

논문은 ALFWorld 성공률 89.9 %를 보고합니다. 우리가 같은 코드로 재현한 값은 79.2 ± 3.3 %입니다. 차이의 원인은 아직 모릅니다.

알아 둘 점

카테고리 판정 규칙은 과제 문장 12종 중 2종을 다른 카테고리로 보냅니다. 예: "put a hot mug in cabinet"은 heat가 아니라 pick_and_place로 갑니다.

EvoSkill (arXiv 2603.02766)

푼 문제

모델을 학습하지 않고 에이전트의 실력을 올리려 합니다. 실패한 실행을 보고 재사용할 수 있는 스킬 폴더를 만듭니다.

방법

세 역할이 돕니다. Executor가 문제를 풉니다. Proposer가 실패를 보고 스킬을 새로 만들거나 고치자고 제안합니다. Skill-Builder가 SKILL.md를 씁니다. 검증 점수가 순위 안에 드는 스킬 묶음만 남깁니다.

스킬을 넣는 방법

Claude Code의 스킬 기능을 그대로 씁니다. 스킬 이름과 설명은 처음부터 system 메시지에 있습니다. 본문은 모델이 Skill tool을 부를 때 대화 중간에 들어옵니다. 대화는 지우지 않고 계속 이어 붙입니다.

성능

OfficeQA 60.6 → 67.9, SealQA 26.6 → 38.7입니다. BrowseComp에 그대로 옮겨도 +5.3입니다.

WikiSkill (arXiv 2608.27454)

푼 문제

스킬을 고칠 때 근거가 흩어져 있는 문제를 풉니다. 실행 기록을 먼저 wiki 페이지로 정리하고, 그 wiki를 근거로 스킬을 고칩니다.

방법

Wiki Maintainer가 실패 패턴 페이지를 씁니다. Skill Proposer가 wiki를 읽고 SKILL.md 수정안 하나를 냅니다. 검증 점수가 지금까지의 최고보다 높을 때만 수정을 남깁니다. 아니면 되돌립니다.

스킬을 넣는 방법

스킬을 고르지 않습니다. 활성 스킬 전체를 Role sentence 바로 뒤에 넣습니다. ALFWorld 템플릿에서는 task 문장이 스킬보다 앞에 있습니다.

알아 둘 점

공식 코드가 없습니다. 실행 모델은 wiki를 보지 않습니다. 우리 실험의 WikiSkill 방식은 논문의 템플릿을 보고 다시 만든 것입니다.

세 시스템을 한 환경에서 비교한 방법WikiSkill과 EvoSkill의 스킬 넣는 방식만 SkillRL의 ALFWorld 환경에 옮겼습니다. 스킬 문서는 SkillRL 스킬 뱅크로 만든 합성 SKILL.md입니다 (2-3절).

1-3 vLLM 기초

prefill과 decode

prefill은 프롬프트 전체를 한 번에 읽고 KV 캐시를 만드는 단계입니다. decode는 응답 토큰을 하나씩 만드는 단계입니다. 재사용은 prefill 계산만 줄입니다.

KV 캐시 용량

vLLM은 시작할 때 GPU 메모리 일부를 KV 캐시용으로 미리 잡습니다. 우리 설정에서는 41.9만 토큰입니다.

해석H100 메모리 80GB의 절반을 vLLM에 주었습니다. 그중 가중치를 뺀 22.36 GiB가 KV 캐시입니다. 토큰 하나의 KV가 약 57 KB라서 캐시에는 41.9만 토큰이 들어갑니다. 2-3절의 빨간 용량선이 이 값입니다.

16토큰 블록

KV 캐시는 16토큰 단위 블록으로 나뉩니다. 재사용도 블록 단위로 합니다.

앞에서부터 같아야 재사용

블록의 이름(해시)은 그 블록과 앞의 모든 블록 내용으로 정해집니다. 그래서 중간에 한 블록이 다르면, 그 뒤 블록은 글이 같아도 재사용하지 못합니다.

초록: 재사용 · 진한 주황: 글이 달라서 다시 계산 · 점선 주황: 글이 같지만 앞 블록이 달라서 다시 계산.

마지막 블록

프롬프트의 마지막 토큰은 항상 새로 계산합니다. 그래서 재사용은 "프롬프트 길이 − 1"을 16의 배수로 내린 값까지입니다.

용량이 찰 때

새 블록이 필요하면 가장 오래 쓰지 않은 블록부터 지웁니다 (LRU). 요청이 끝나도 블록은 지워지기 전까지 재사용할 수 있습니다. vLLM은 지운 블록 수를 기록하지 않습니다.

선점

실행 중인 요청의 블록이 모자라면 가장 늦게 들어온 요청을 멈추고, 나중에 처음부터 다시 계산합니다.

1-1절의 step 2 요청은 1,421토큰이고 928토큰을 캐시에서 가져왔습니다. 직전 요청과는 "Prior to this step, you have already taken N step(s)"의 숫자 N 앞까지 같습니다. 그 지점을 16의 배수로 내린 값이 928입니다.

2-0 측정 방법

• SkillRL의 공식 실행 경로(verl)는 스텝마다 캐시를 비웁니다. 그래서 verl 없이 같은 프롬프트를 만드는 실행기(하네스)를 따로 만들었습니다.
• 하네스는 공식 경로와 프롬프트 1,259개가 글자 단위로 같았습니다. 성공률은 80.2 ± 2.4 %로 공식 경로의 79.2 ± 3.3 %와 같은 범위였습니다.
• 재사용률은 vLLM이 요청마다 보고한 "캐시에서 가져온 토큰 수"로 잽니다.
• 조건 하나는 게임 64개 × seed 3개입니다. seed는 모델이 응답을 고를 때 쓰는 난수의 시작값입니다 (1-1절). seed가 3개뿐이라서 8~10 %p보다 작은 성공률 차이는 우연과 구별하지 못합니다.

2-1 재사용이 끊기는 곳

해석가로축은 재사용률입니다 (0~1). 공식 경로처럼 스텝마다 캐시를 비우면 프롬프트의 16 %만 캐시에서 가져옵니다. 캐시를 스텝 사이에 남겨 두면 63 %를 가져옵니다. 같은 프롬프트, 같은 모델이고 캐시를 지우는지만 다릅니다.

• 캐시를 스텝 사이에 남겨 두면 재사용률이 0.160에서 0.633으로 오릅니다.
• 같은 에피소드의 재사용은 "## Current Progress" 바로 뒤의 스텝 숫자에서 끊깁니다. 그 앞부분이 프롬프트의 0.637~0.643입니다.
• 그래서 이 형식에서 재사용률의 실질 상한은 약 0.64입니다.

2-2 순서를 바꾸면 얼마나 줄고, 무엇을 잃나

정적 분석: 다른 에피소드와 공유하는 앞부분

해석막대는 두 에피소드의 프롬프트가 처음부터 몇 토큰까지 같은지를 나타냅니다. 길수록 다른 에피소드가 만든 캐시를 많이 쓸 수 있습니다. 원래 순서는 44토큰에서 끝납니다. 두 번째 줄의 task 문장이 에피소드마다 다르기 때문입니다. 스킬을 task 앞으로 옮기면 같은 카테고리끼리 920토큰, 곧 프롬프트의 87 %가 같습니다. 다른 카테고리끼리는 카테고리 스킬에서 갈라져서 263토큰 또는 541토큰입니다.

두 에피소드의 프롬프트를 앞에서부터 비교

G = General Principles, C = 카테고리 스킬, M = Mistakes to Avoid. 에피소드 B의 색은 A와 비교한 결과입니다.

원래 순서에서는 task 문장이 스킬 앞에 있습니다. 그래서 다른 에피소드와는 44토큰만 같습니다. 스킬을 task 문장 앞으로 옮기면 같은 카테고리끼리 920토큰이 같아집니다.

GPU 실험: 순서를 바꿔도 성공률이 유지되나

해석점이 세로 점선(원래 순서 79.2 %)에 가까우면 성공률이 유지된 것입니다. 가로선은 seed 3개의 표준편차입니다. 위의 세 형식은 점선과 2 %p 안에 있습니다. 이동 목록 분리만 31.2 %로 크게 떨어졌습니다.

형식별 user 메시지 구조 (step 1 이후)

• A: 같은 카테고리의 다른 에피소드와 같은 부분이 여기서 끝납니다. 공통 먼저 형식에서 다른 카테고리끼리는 M 뒤에서 끝납니다.
• B: 같은 에피소드의 다음 스텝과 같은 부분이 여기서 끝납니다. B 뒤는 매 스텝 다시 계산합니다.

출처: EXPERIMENT_PLAN.md 3.3 1차 결과.

• 구간 순서와 지시문 위치를 바꾼 세 형식은 성공률 차이가 허용 범위 안입니다.
• 행동 목록의 모양을 바꾼 형식은 성공률이 79.2 %에서 31.2 %로 떨어졌습니다. 모델이 새 장소로 거의 가지 않았습니다 (에피소드당 9.6회 → 3.7회).

2-3 세 시스템 방식 × 모델 세 개 (GPU)

형식별 프롬프트 예시

다섯 형식에서 같은 게임의 같은 스텝 요청을 하나씩 골랐습니다. 모델은 RL 모델이고 seed는 0입니다. 게임은 env 0이고 과제는 "put a clean cloth in cabinet"이다 (clean 카테고리). 요청은 step 2 요청입니다.

• 형식마다 run이 다릅니다. 그래서 이력의 내용이 형식마다 다릅니다.
• 구간별 토큰 수의 합은 다섯 요청 모두 덤프의 프롬프트 토큰 수와 같습니다.

해석막대 하나가 요청 하나입니다. 왼쪽 끝이 프롬프트의 처음입니다. 파란 스킬 구간의 크기와 위치가 형식마다 다릅니다. SkillRL 원래는 스킬이 871토큰이고 프롬프트 가운데에 있습니다. WikiSkill 두 방식은 스킬 문서 전체를 넣어서 프롬프트의 대부분이 스킬입니다. EvoSkill 두 방식은 tool 결과로 들어온 스킬 문서가 이전 대화(주황) 사이에 있습니다.

형식을 고르면 아래에 그 요청의 실제 프롬프트가 나옵니다. 배경색은 막대의 구간 색과 같습니다. 글자에 포인터를 올리면 구간 이름과 토큰 수가 나옵니다.

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: put a clean cloth in cabinet.

## Retrieved Relevant Experience

### General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.

### Clean Skills
- **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
  _Apply when: As soon as the goal specifies the object must be clean before placement._
- **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
  _Apply when: After visually confirming the presence of the target object on any surface or inside an opened container._
- **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
  _Apply when: Once the target object is in hand and its required state is ‘clean’._
- **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
  _Apply when: After initial obvious surfaces are empty and the target object remains unfound._
- **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
  _Apply when: Immediately after a cleaning action and before placing the object at its target location._
- **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.
  _Apply when: At the very start of the task to choose the first search destination for the target object._

### Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.

## Current Progress

Prior to this step, you have already taken 2 step(s). Below are the most recent 2 observations and the corresponding actions you took: [Observation 1: '-= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.', Action 1: 'go to handtowelholder 1']
[Observation 2: 'You arrive at handtowelholder 1. On the handtowelholder 1, you see a handtowel 1.', Action 2: 'take handtowel 1 from handtowelholder 1']
You are now at step 3 and your current observation is: You pick up the handtowel 1 from the handtowelholder 1.
Your admissible actions of the current situation are: ['examine handtowel 1'
 'examine handtowelholder 1'
 'go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'move handtowel 1 to handtowelholder 1'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment. Your task is to: put a clean cloth in cabinet.

---
name: alfworld-core
description: Exploration principles and mistakes for every ALFWorld task.
---

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
  _When: Upon first visual confirmation of a goal-relevant object._
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
  _When: Holding any goal object while its target location has been identified._
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
  _When: Throughout multi-instance tasks (e.g., “put two X…”)._
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
  _When: After picking up an object that must change temperature or cleanliness._
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.
  _When: Tasks containing spatial prepositions like under, inside, on, in_front_of._
- **Open Before Judging Empty**: Treat any closed container as unexplored; open it and inspect contents before deciding that the goal object is absent.
  _When: Encountering a closed drawer, cabinet, fridge, microwave, etc._
- **Avoid Redundant Rechecks**: Record which locations and containers have been fully inspected; do not revisit them unless new evidence suggests their contents changed.
  _When: After the first full pass of a location or when considering next search target._
- **Plan with Landmarks**: Continuously update and reference a map of known object and container locations to choose shortest paths and prevent aimless wandering.
  _When: Each time a new furniture piece, container, or goal object is observed._
- **Progressive Goal Decomposition**: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination/reference object, (4) satisfy spatial relation/placement, completing each sub-goal before moving to the next.
  _When: At the start of every episode and whenever the environment state invalidates the current sub-goal._
- **Efficient Relation Search**: When the goal pairs two objects (A under/on/inside B), restrict search for A to locations near B and vice-versa, instead of treating them independently.
  _When: On goals that mention both a target object and a reference object._
- **Terminate Only On Success**: Issue the 'done' action exclusively after validating that all goal conditions are met; otherwise continue searching or correcting object states.
  _When: Before ending an episode or when the agent believes the task is complete._

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.
- **Don't**: Agent completes spatial placement while skipping required state changes (e.g., cleaning, heating).
  **Instead**: Integrate state precondition checks into the planner so that placement actions are only considered valid after verifying and, if necessary, executing the requisite state-changing operations.
- **Don't**: Agent repeatedly toggles or examines objects that yield no new information or progress.
  **Instead**: Implement novelty or information gain metrics that discourage repeating identical actions unless the environment state has changed since the last attempt.
- **Don't**: Agent mishandles spatial relations (e.g., ‘under’, ‘inside’, ‘behind’) by inspecting only obvious surfaces and ignoring less visible sub-locations.
  **Instead**: During task parsing, generate explicit search targets for all sub-locations associated with the reference object (drawers, shelves, underside) and iterate through them systematically.
- **Don't**: Agent forgets which object it is holding, leading to unnecessary pickups, drops, or misplacements.
  **Instead**: Maintain a dedicated ‘hand inventory’ structure updated after every pickup/drop, and consult it before issuing navigation or interaction commands.
- **Don't**: Agent executes inefficient action sequences by searching for each required object serially rather than batching placements or leveraging proximity.
  **Instead**: Use planning algorithms that group similar subtasks, allowing the agent to collect all required objects in one pass before performing consolidated delivery or transformation steps.
- **Don't**: Agent prematurely terminates or stalls without having observed any candidate objects that satisfy the goal.
  **Instead**: Only allow ‘done’ actions when explicit goal predicates are met; otherwise, continue targeted exploration of remaining unvisited rooms and containers.

---
name: alfworld-tasks
description: Step-by-step instructions per ALFWorld task type.
---

## Pick and Place Instructions
1. **Systematic First-Pass Search**: Maintain a checklist of all visible and closed containers/surfaces; open or inspect each unseen candidate exactly once before revisiting any location.
2. **Grab When Seen**: Whenever a needed object is visible and reachable, immediately take it before moving elsewhere—do not leave targets uncollected.
3. **Transform Before Transport**: If the goal specifies a state change (e.g., heated, cooled, cleaned), perform the transformation at the nearest appropriate appliance before heading to the final destination.
4. **Place Before More Search**: When holding any goal object and the target location is known and reachable, navigate there and place it immediately, then resume searching if more items are needed.
5. **Track Counts for Multiples**: Maintain a tally of how many target objects are required, held, and already placed; stop searching only after the placed count meets the goal.

## Look at Object in Light Instructions
1. **Seek Lamp Surfaces First**: Head straight to furniture that commonly hosts a desklamp (desk, sidetable, nightstand) because the target must end up under that light.
2. **Switch Lamp On**: Issue 'use desklamp' as soon as you reach it so the light condition is satisfied before or immediately after handling the target object.
3. **Acquire Or Fetch Target**: If the target is on the lit surface, 'take' it; if found elsewhere, pick it up and carry it back to the lit lamp area.
4. **Systematic Surface Search**: On each candidate lamp surface, look, open nearby containers/drawers once, then move on—ensuring all plausible lamp locations are covered.
5. **Prevent Action Loops**: Avoid toggling the same lamp or re-examining the same empty spot; if no progress after one full check, relocate to a new plausible lamp location.

## Clean Instructions
1. **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
2. **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
4. **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
5. **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
6. **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.

## Heat Instructions
1. **Secure Exact Target First**: Always identify and pick up the exact object type named in the goal before interacting with the microwave or destination, ignoring look-alikes (e.g., don’t substitute a mug for a cup).
2. **Systematic One-Pass Search**: Search each plausible surface or closed container once by opening it and inspecting contents; mark searched spots mentally to avoid redundant revisits.
3. **Open Then Heat**: Upon reaching the microwave with the target in hand, always open the door, place the object inside, and execute the heat action before leaving.
4. **No Appliance Before Object**: Do not move to or interact with the microwave or final placement location until the target object is picked up, preventing wasted navigation steps.
5. **Direct Post-Heat Placement**: After heating, navigate straight to the specified destination and place the object once, avoiding extra exploration or detours.

## Cool Instructions
1. **Systematic Object Hunt**: Sweep every unvisited surface and open each closed container until the exact required object is found; never loop over already-checked empty spots.
2. **Confirm Object Match**: After picking up an item, verify it matches the requested type; if not, drop it and resume the search.
3. **Prep Cooling Appliance**: Locate the fridge first and open it so it’s ready for use before or immediately after grabbing the target object.
4. **Enforce Cooling Before Placement**: Do not place the target object in its final location until you have successfully executed a cooling action with the fridge.
5. **Direct Post-Cooling Delivery**: Once the object is cooled, navigate straight to the destination and place it without extraneous detours or re-searching.

## Examine Instructions
1. **Pair Objects Early**: When both the target and desklamp are seen on the same surface, immediately pick up (or leave) the target there and use the desklamp without moving elsewhere.
2. **Grab Target First**: If you see the target but not the desklamp yet, take the target immediately so you can carry it to wherever the desklamp is found.
3. **Shortcut To Tool**: Once holding the target, navigate straight to the nearest known desklamp and issue a single USE command—no intervening searches or detours.
4. **Single Toggle Rule**: Use (toggle) the desklamp only after the target is in hand or co-located; avoid repeated or premature toggles that waste steps.
5. **Explore New Surfaces**: If either the target or the desklamp hasn’t been found, systematically check unvisited surfaces/containers instead of re-inspecting places already confirmed empty.
6. **Lock In Observed Items**: After spotting either required object, do not leave without acting—pick it up (if target) or note its location (if desklamp) to prevent costly backtracking loops.

## Current Progress

Prior to this step, you have already taken 2 step(s). Below are the most recent 2 observations and the corresponding actions you took: [Observation 1: '-= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.', Action 1: 'go to countertop 1']
[Observation 2: 'You arrive at countertop 1. On the countertop 1, you see a candle 2.', Action 2: 'examine countertop 1']
You are now at step 3 and your current observation is: On the countertop 1, you see a candle 2.
Your admissible actions of the current situation are: ['examine countertop 1'
 'go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'take candle 2 from countertop 1'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment.

---
name: alfworld-core
description: Exploration principles and mistakes for every ALFWorld task.
---

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
  _When: Upon first visual confirmation of a goal-relevant object._
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
  _When: Holding any goal object while its target location has been identified._
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
  _When: Throughout multi-instance tasks (e.g., “put two X…”)._
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
  _When: After picking up an object that must change temperature or cleanliness._
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.
  _When: Tasks containing spatial prepositions like under, inside, on, in_front_of._
- **Open Before Judging Empty**: Treat any closed container as unexplored; open it and inspect contents before deciding that the goal object is absent.
  _When: Encountering a closed drawer, cabinet, fridge, microwave, etc._
- **Avoid Redundant Rechecks**: Record which locations and containers have been fully inspected; do not revisit them unless new evidence suggests their contents changed.
  _When: After the first full pass of a location or when considering next search target._
- **Plan with Landmarks**: Continuously update and reference a map of known object and container locations to choose shortest paths and prevent aimless wandering.
  _When: Each time a new furniture piece, container, or goal object is observed._
- **Progressive Goal Decomposition**: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination/reference object, (4) satisfy spatial relation/placement, completing each sub-goal before moving to the next.
  _When: At the start of every episode and whenever the environment state invalidates the current sub-goal._
- **Efficient Relation Search**: When the goal pairs two objects (A under/on/inside B), restrict search for A to locations near B and vice-versa, instead of treating them independently.
  _When: On goals that mention both a target object and a reference object._
- **Terminate Only On Success**: Issue the 'done' action exclusively after validating that all goal conditions are met; otherwise continue searching or correcting object states.
  _When: Before ending an episode or when the agent believes the task is complete._

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.
- **Don't**: Agent completes spatial placement while skipping required state changes (e.g., cleaning, heating).
  **Instead**: Integrate state precondition checks into the planner so that placement actions are only considered valid after verifying and, if necessary, executing the requisite state-changing operations.
- **Don't**: Agent repeatedly toggles or examines objects that yield no new information or progress.
  **Instead**: Implement novelty or information gain metrics that discourage repeating identical actions unless the environment state has changed since the last attempt.
- **Don't**: Agent mishandles spatial relations (e.g., ‘under’, ‘inside’, ‘behind’) by inspecting only obvious surfaces and ignoring less visible sub-locations.
  **Instead**: During task parsing, generate explicit search targets for all sub-locations associated with the reference object (drawers, shelves, underside) and iterate through them systematically.
- **Don't**: Agent forgets which object it is holding, leading to unnecessary pickups, drops, or misplacements.
  **Instead**: Maintain a dedicated ‘hand inventory’ structure updated after every pickup/drop, and consult it before issuing navigation or interaction commands.
- **Don't**: Agent executes inefficient action sequences by searching for each required object serially rather than batching placements or leveraging proximity.
  **Instead**: Use planning algorithms that group similar subtasks, allowing the agent to collect all required objects in one pass before performing consolidated delivery or transformation steps.
- **Don't**: Agent prematurely terminates or stalls without having observed any candidate objects that satisfy the goal.
  **Instead**: Only allow ‘done’ actions when explicit goal predicates are met; otherwise, continue targeted exploration of remaining unvisited rooms and containers.

---
name: alfworld-tasks
description: Step-by-step instructions per ALFWorld task type.
---

## Pick and Place Instructions
1. **Systematic First-Pass Search**: Maintain a checklist of all visible and closed containers/surfaces; open or inspect each unseen candidate exactly once before revisiting any location.
2. **Grab When Seen**: Whenever a needed object is visible and reachable, immediately take it before moving elsewhere—do not leave targets uncollected.
3. **Transform Before Transport**: If the goal specifies a state change (e.g., heated, cooled, cleaned), perform the transformation at the nearest appropriate appliance before heading to the final destination.
4. **Place Before More Search**: When holding any goal object and the target location is known and reachable, navigate there and place it immediately, then resume searching if more items are needed.
5. **Track Counts for Multiples**: Maintain a tally of how many target objects are required, held, and already placed; stop searching only after the placed count meets the goal.

## Look at Object in Light Instructions
1. **Seek Lamp Surfaces First**: Head straight to furniture that commonly hosts a desklamp (desk, sidetable, nightstand) because the target must end up under that light.
2. **Switch Lamp On**: Issue 'use desklamp' as soon as you reach it so the light condition is satisfied before or immediately after handling the target object.
3. **Acquire Or Fetch Target**: If the target is on the lit surface, 'take' it; if found elsewhere, pick it up and carry it back to the lit lamp area.
4. **Systematic Surface Search**: On each candidate lamp surface, look, open nearby containers/drawers once, then move on—ensuring all plausible lamp locations are covered.
5. **Prevent Action Loops**: Avoid toggling the same lamp or re-examining the same empty spot; if no progress after one full check, relocate to a new plausible lamp location.

## Clean Instructions
1. **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
2. **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
4. **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
5. **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
6. **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.

## Heat Instructions
1. **Secure Exact Target First**: Always identify and pick up the exact object type named in the goal before interacting with the microwave or destination, ignoring look-alikes (e.g., don’t substitute a mug for a cup).
2. **Systematic One-Pass Search**: Search each plausible surface or closed container once by opening it and inspecting contents; mark searched spots mentally to avoid redundant revisits.
3. **Open Then Heat**: Upon reaching the microwave with the target in hand, always open the door, place the object inside, and execute the heat action before leaving.
4. **No Appliance Before Object**: Do not move to or interact with the microwave or final placement location until the target object is picked up, preventing wasted navigation steps.
5. **Direct Post-Heat Placement**: After heating, navigate straight to the specified destination and place the object once, avoiding extra exploration or detours.

## Cool Instructions
1. **Systematic Object Hunt**: Sweep every unvisited surface and open each closed container until the exact required object is found; never loop over already-checked empty spots.
2. **Confirm Object Match**: After picking up an item, verify it matches the requested type; if not, drop it and resume the search.
3. **Prep Cooling Appliance**: Locate the fridge first and open it so it’s ready for use before or immediately after grabbing the target object.
4. **Enforce Cooling Before Placement**: Do not place the target object in its final location until you have successfully executed a cooling action with the fridge.
5. **Direct Post-Cooling Delivery**: Once the object is cooled, navigate straight to the destination and place it without extraneous detours or re-searching.

## Examine Instructions
1. **Pair Objects Early**: When both the target and desklamp are seen on the same surface, immediately pick up (or leave) the target there and use the desklamp without moving elsewhere.
2. **Grab Target First**: If you see the target but not the desklamp yet, take the target immediately so you can carry it to wherever the desklamp is found.
3. **Shortcut To Tool**: Once holding the target, navigate straight to the nearest known desklamp and issue a single USE command—no intervening searches or detours.
4. **Single Toggle Rule**: Use (toggle) the desklamp only after the target is in hand or co-located; avoid repeated or premature toggles that waste steps.
5. **Explore New Surfaces**: If either the target or the desklamp hasn’t been found, systematically check unvisited surfaces/containers instead of re-inspecting places already confirmed empty.
6. **Lock In Observed Items**: After spotting either required object, do not leave without acting—pick it up (if target) or note its location (if desklamp) to prevent costly backtracking loops.

Your task is to: put a clean cloth in cabinet.

## Current Progress

Prior to this step, you have already taken 2 step(s). Below are the most recent 2 observations and the corresponding actions you took: [Observation 1: '-= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.', Action 1: 'go to countertop 1']
[Observation 2: 'You arrive at countertop 1. On the countertop 1, you see a candle 2.', Action 2: 'examine countertop 1']
You are now at step 3 and your current observation is: On the countertop 1, you see a candle 2.
Your admissible actions of the current situation are: ['examine countertop 1'
 'go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'take candle 2 from countertop 1'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant

<|im_start|>system
You are an expert agent operating in the ALFRED Embodied Environment.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

Available skills (load one with the Skill tool when it applies):
- alfworld-pick-and-place: How to solve ALFWorld Pick and Place tasks.
- alfworld-look-at-obj-in-light: How to solve ALFWorld Look at Object in Light tasks.
- alfworld-clean: How to solve ALFWorld Clean tasks.
- alfworld-heat: How to solve ALFWorld Heat tasks.
- alfworld-cool: How to solve ALFWorld Cool tasks.
- alfworld-examine: How to solve ALFWorld Examine tasks.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment.
Your current observation is: -= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.
Your admissible actions of the current situation are: ['go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. 
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant
<think>I need to locate a cloth first. I’ll start by checking the most likely storage place for towels and hand towels: the bathtub basin.</think>
<action>go to bathtubbasin 1</action><|im_end|>
<|im_start|>user
<tool_response>
---
name: alfworld-clean
description: How to solve ALFWorld Clean tasks.
---

## When to Apply
- As soon as the goal specifies the object must be clean before placement.
- After visually confirming the presence of the target object on any surface or inside an opened container.
- Once the target object is in hand and its required state is ‘clean’.
- After initial obvious surfaces are empty and the target object remains unfound.
- Immediately after a cleaning action and before placing the object at its target location.
- At the very start of the task to choose the first search destination for the target object.

## Clean Instructions
1. **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
2. **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
4. **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
5. **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
6. **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
  _When: Upon first visual confirmation of a goal-relevant object._
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
  _When: Holding any goal object while its target location has been identified._
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
  _When: Throughout multi-instance tasks (e.g., “put two X…”)._
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
  _When: After picking up an object that must change temperature or cleanliness._
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.
  _When: Tasks containing spatial prepositions like under, inside, on, in_front_of._
- **Open Before Judging Empty**: Treat any closed container as unexplored; open it and inspect contents before deciding that the goal object is absent.
  _When: Encountering a closed drawer, cabinet, fridge, microwave, etc._
- **Avoid Redundant Rechecks**: Record which locations and containers have been fully inspected; do not revisit them unless new evidence suggests their contents changed.
  _When: After the first full pass of a location or when considering next search target._
- **Plan with Landmarks**: Continuously update and reference a map of known object and container locations to choose shortest paths and prevent aimless wandering.
  _When: Each time a new furniture piece, container, or goal object is observed._
- **Progressive Goal Decomposition**: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination/reference object, (4) satisfy spatial relation/placement, completing each sub-goal before moving to the next.
  _When: At the start of every episode and whenever the environment state invalidates the current sub-goal._
- **Efficient Relation Search**: When the goal pairs two objects (A under/on/inside B), restrict search for A to locations near B and vice-versa, instead of treating them independently.
  _When: On goals that mention both a target object and a reference object._
- **Terminate Only On Success**: Issue the 'done' action exclusively after validating that all goal conditions are met; otherwise continue searching or correcting object states.
  _When: Before ending an episode or when the agent believes the task is complete._

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.
- **Don't**: Agent completes spatial placement while skipping required state changes (e.g., cleaning, heating).
  **Instead**: Integrate state precondition checks into the planner so that placement actions are only considered valid after verifying and, if necessary, executing the requisite state-changing operations.
- **Don't**: Agent repeatedly toggles or examines objects that yield no new information or progress.
  **Instead**: Implement novelty or information gain metrics that discourage repeating identical actions unless the environment state has changed since the last attempt.
- **Don't**: Agent mishandles spatial relations (e.g., ‘under’, ‘inside’, ‘behind’) by inspecting only obvious surfaces and ignoring less visible sub-locations.
  **Instead**: During task parsing, generate explicit search targets for all sub-locations associated with the reference object (drawers, shelves, underside) and iterate through them systematically.
- **Don't**: Agent forgets which object it is holding, leading to unnecessary pickups, drops, or misplacements.
  **Instead**: Maintain a dedicated ‘hand inventory’ structure updated after every pickup/drop, and consult it before issuing navigation or interaction commands.
- **Don't**: Agent executes inefficient action sequences by searching for each required object serially rather than batching placements or leveraging proximity.
  **Instead**: Use planning algorithms that group similar subtasks, allowing the agent to collect all required objects in one pass before performing consolidated delivery or transformation steps.
- **Don't**: Agent prematurely terminates or stalls without having observed any candidate objects that satisfy the goal.
  **Instead**: Only allow ‘done’ actions when explicit goal predicates are met; otherwise, continue targeted exploration of remaining unvisited rooms and containers.
</tool_response><|im_end|>
<|im_start|>user
Observation: You arrive at bathtubbasin 1. On the bathtubbasin 1, you see a cloth 2, and a soapbar 1.
Admissible actions: ['examine bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'take cloth 2 from bathtubbasin 1'
 'take soapbar 1 from bathtubbasin 1'].<|im_end|>
<|im_start|>assistant
<think>I need to pick up a cloth. I’ll start by checking the bathtub basin.</think>
<action>take cloth 2 from bathtubbasin 1</action><|im_end|>
<|im_start|>user
Observation: You pick up the cloth 2 from the bathtubbasin 1.
Admissible actions: ['examine bathtubbasin 1'
 'examine cloth 2'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'move cloth 2 to bathtubbasin 1'].<|im_end|>
<|im_start|>assistant

<|im_start|>system
You are an expert agent operating in the ALFRED Embodied Environment.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.

Available skills (load one with the Skill tool when it applies):
- alfworld-pick-and-place: How to solve ALFWorld Pick and Place tasks.
- alfworld-look-at-obj-in-light: How to solve ALFWorld Look at Object in Light tasks.
- alfworld-clean: How to solve ALFWorld Clean tasks.
- alfworld-heat: How to solve ALFWorld Heat tasks.
- alfworld-cool: How to solve ALFWorld Cool tasks.
- alfworld-examine: How to solve ALFWorld Examine tasks.

---
name: alfworld-core
description: Principles and mistakes for every ALFWorld task (always loaded).
---

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
  _When: Upon first visual confirmation of a goal-relevant object._
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
  _When: Holding any goal object while its target location has been identified._
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
  _When: Throughout multi-instance tasks (e.g., “put two X…”)._
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
  _When: After picking up an object that must change temperature or cleanliness._
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.
  _When: Tasks containing spatial prepositions like under, inside, on, in_front_of._
- **Open Before Judging Empty**: Treat any closed container as unexplored; open it and inspect contents before deciding that the goal object is absent.
  _When: Encountering a closed drawer, cabinet, fridge, microwave, etc._
- **Avoid Redundant Rechecks**: Record which locations and containers have been fully inspected; do not revisit them unless new evidence suggests their contents changed.
  _When: After the first full pass of a location or when considering next search target._
- **Plan with Landmarks**: Continuously update and reference a map of known object and container locations to choose shortest paths and prevent aimless wandering.
  _When: Each time a new furniture piece, container, or goal object is observed._
- **Progressive Goal Decomposition**: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination/reference object, (4) satisfy spatial relation/placement, completing each sub-goal before moving to the next.
  _When: At the start of every episode and whenever the environment state invalidates the current sub-goal._
- **Efficient Relation Search**: When the goal pairs two objects (A under/on/inside B), restrict search for A to locations near B and vice-versa, instead of treating them independently.
  _When: On goals that mention both a target object and a reference object._
- **Terminate Only On Success**: Issue the 'done' action exclusively after validating that all goal conditions are met; otherwise continue searching or correcting object states.
  _When: Before ending an episode or when the agent believes the task is complete._

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.
- **Don't**: Agent completes spatial placement while skipping required state changes (e.g., cleaning, heating).
  **Instead**: Integrate state precondition checks into the planner so that placement actions are only considered valid after verifying and, if necessary, executing the requisite state-changing operations.
- **Don't**: Agent repeatedly toggles or examines objects that yield no new information or progress.
  **Instead**: Implement novelty or information gain metrics that discourage repeating identical actions unless the environment state has changed since the last attempt.
- **Don't**: Agent mishandles spatial relations (e.g., ‘under’, ‘inside’, ‘behind’) by inspecting only obvious surfaces and ignoring less visible sub-locations.
  **Instead**: During task parsing, generate explicit search targets for all sub-locations associated with the reference object (drawers, shelves, underside) and iterate through them systematically.
- **Don't**: Agent forgets which object it is holding, leading to unnecessary pickups, drops, or misplacements.
  **Instead**: Maintain a dedicated ‘hand inventory’ structure updated after every pickup/drop, and consult it before issuing navigation or interaction commands.
- **Don't**: Agent executes inefficient action sequences by searching for each required object serially rather than batching placements or leveraging proximity.
  **Instead**: Use planning algorithms that group similar subtasks, allowing the agent to collect all required objects in one pass before performing consolidated delivery or transformation steps.
- **Don't**: Agent prematurely terminates or stalls without having observed any candidate objects that satisfy the goal.
  **Instead**: Only allow ‘done’ actions when explicit goal predicates are met; otherwise, continue targeted exploration of remaining unvisited rooms and containers.<|im_end|>
<|im_start|>user

You are an expert agent operating in the ALFRED Embodied Environment.
Your current observation is: -= Welcome to TextWorld, ALFRED! =-

You are in the middle of a room. Looking quickly around you, you see a bathtubbasin 1, a cabinet 5, a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.

Your task is to: put a clean cloth in cabinet.
Your admissible actions of the current situation are: ['go to bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'].

Now it's your turn to take an action.
You should first reason step-by-step about the current situation. This reasoning process MUST be enclosed within <think> </think> tags. 
Once you've finished your reasoning, you should choose an admissible action for current step and present it within <action> </action> tags.
<|im_end|>
<|im_start|>assistant
<think>I need to locate a cloth first. I’ll start by checking the most common storage place for towels and hand towels: the bathtub basin.</think>
<action>go to bathtubbasin 1</action><|im_end|>
<|im_start|>user
<tool_response>
---
name: alfworld-clean
description: How to solve ALFWorld Clean tasks.
---

## When to Apply
- As soon as the goal specifies the object must be clean before placement.
- After visually confirming the presence of the target object on any surface or inside an opened container.
- Once the target object is in hand and its required state is ‘clean’.
- After initial obvious surfaces are empty and the target object remains unfound.
- Immediately after a cleaning action and before placing the object at its target location.
- At the very start of the task to choose the first search destination for the target object.

## Clean Instructions
1. **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
2. **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
4. **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
5. **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
6. **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.
</tool_response><|im_end|>
<|im_start|>user
Observation: You arrive at bathtubbasin 1. On the bathtubbasin 1, you see a cloth 2, and a soapbar 1.
Admissible actions: ['examine bathtubbasin 1'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'take cloth 2 from bathtubbasin 1'
 'take soapbar 1 from bathtubbasin 1'].<|im_end|>
<|im_start|>assistant
<think>I see a cloth on the bathtub basin. I should pick it up right away (Immediate Acquisition).</think>
<action>take cloth 2 from bathtubbasin 1</action><|im_end|>
<|im_start|>user
Observation: You pick up the cloth 2 from the bathtubbasin 1.
Admissible actions: ['examine bathtubbasin 1'
 'examine cloth 2'
 'go to cabinet 1'
 'go to cabinet 2'
 'go to cabinet 3'
 'go to cabinet 4'
 'go to cabinet 5'
 'go to countertop 1'
 'go to garbagecan 1'
 'go to handtowelholder 1'
 'go to handtowelholder 2'
 'go to sinkbasin 1'
 'go to toilet 1'
 'go to toiletpaperhanger 1'
 'go to towelholder 1'
 'inventory'
 'look'
 'move cloth 2 to bathtubbasin 1'].<|im_end|>
<|im_start|>assistant

{
  "skill_id": "gen_001",
  "title": "Systematic Exploration",
  "principle": "Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.",
  "when_to_apply": "Anytime the goal object count is not yet met and unexplored locations remain."
}

- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._

{
  "skill_id": "cle_003",
  "title": "Sink First for Cleaning",
  "principle": "Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.",
  "when_to_apply": "Once the target object is in hand and its required state is ‘clean’."
}

- **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
  _Apply when: Once the target object is in hand and its required state is ‘clean’._

## Clean Instructions
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.

## When to Apply
- Once the target object is in hand and its required state is ‘clean’.

{
  "mistake_id": "err_001",
  "description": "Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.",
  "why_it_happens": "The agent lacks an explicit memory or policy for tracking explored areas, so its search strategy degenerates into local loops.",
  "how_to_avoid": "Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted."
}

- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.

---
name: alfworld-clean
description: How to solve ALFWorld Clean tasks.
---

## When to Apply
- As soon as the goal specifies the object must be clean before placement.
- After visually confirming the presence of the target object on any surface or inside an opened container.
- Once the target object is in hand and its required state is ‘clean’.
- After initial obvious surfaces are empty and the target object remains unfound.
- Immediately after a cleaning action and before placing the object at its target location.
- At the very start of the task to choose the first search destination for the target object.

## Clean Instructions
1. **Phase-Ordered Plan**: Always execute clean tasks in the fixed sequence: 1) locate & acquire target object, 2) bring it to an available water source/sink to clean, 3) navigate to final location, 4) place object.
2. **Pick Before You Wander**: If the target object is visible or discovered, take it immediately; never leave it behind to explore other places, as possession enables direct cleaning and prevents redundant searches.
3. **Sink First for Cleaning**: Upon holding the target object, go straight to the nearest sink, basin, or faucet and issue the clean/use command before any placement attempts.
4. **Systematic Container Sweep**: When the object isn’t found on open surfaces, iterate through all unopened or unexamined containers in the room before revisiting already-checked spots to avoid search loops.
5. **State Verification Before Drop**: Always inspect or infer the object’s state after cleaning; if still not confirmed clean, re-clean before final placement to satisfy goal conditions.
6. **Use Location Priors**: Begin search at the most probable category-specific surfaces (e.g., kitchenware on countertop/stove, food on dining table) to minimize exploration steps and avoid aimless roaming.

## General Principles
- **Systematic Exploration**: Search every plausible surface or container exactly once before revisiting; prioritize unopened or unseen locations to cover the whole room methodically.
  _When: Anytime the goal object count is not yet met and unexplored locations remain._
- **Immediate Acquisition**: As soon as a required object becomes visible and reachable, take it before moving elsewhere to avoid losing track or re-searching.
  _When: Upon first visual confirmation of a goal-relevant object._
- **Destination First Policy**: After picking up a goal object, navigate directly to the known target receptacle and place it before resuming further search.
  _When: Holding any goal object while its target location has been identified._
- **Track Counts & Progress**: Maintain an internal counter of how many goal objects are still needed; stop searching or terminate episode only when the counter reaches zero.
  _When: Throughout multi-instance tasks (e.g., “put two X…”)._
- **Use State-Changing Tools Early**: For tasks requiring heated/cooled/clean objects, acquire the object, then immediately use the nearest suitable appliance to change its state before any placement.
  _When: After picking up an object that must change temperature or cleanliness._
- **Establish Spatial Relations**: When a goal specifies a relation (under/inside/on), first locate the reference object, adjust its state if needed (e.g., turn lamp on), then search the specified spatial region for the target object or place it there.
  _When: Tasks containing spatial prepositions like under, inside, on, in_front_of._
- **Open Before Judging Empty**: Treat any closed container as unexplored; open it and inspect contents before deciding that the goal object is absent.
  _When: Encountering a closed drawer, cabinet, fridge, microwave, etc._
- **Avoid Redundant Rechecks**: Record which locations and containers have been fully inspected; do not revisit them unless new evidence suggests their contents changed.
  _When: After the first full pass of a location or when considering next search target._
- **Plan with Landmarks**: Continuously update and reference a map of known object and container locations to choose shortest paths and prevent aimless wandering.
  _When: Each time a new furniture piece, container, or goal object is observed._
- **Progressive Goal Decomposition**: Break tasks into ordered sub-goals: (1) locate object, (2) obtain or transform it, (3) locate destination/reference object, (4) satisfy spatial relation/placement, completing each sub-goal before moving to the next.
  _When: At the start of every episode and whenever the environment state invalidates the current sub-goal._
- **Efficient Relation Search**: When the goal pairs two objects (A under/on/inside B), restrict search for A to locations near B and vice-versa, instead of treating them independently.
  _When: On goals that mention both a target object and a reference object._
- **Terminate Only On Success**: Issue the 'done' action exclusively after validating that all goal conditions are met; otherwise continue searching or correcting object states.
  _When: Before ending an episode or when the agent believes the task is complete._

## Mistakes to Avoid
- **Don't**: Agent repeatedly revisits the same already-searched locations or containers instead of exploring new ones.
  **Instead**: Maintain an exploration map that records which rooms/containers have been fully inspected and prioritize unvisited or unopened candidates until all reasonable options are exhausted.
- **Don't**: Agent decides a closed container is empty without opening it, causing missed discoveries of target objects.
  **Instead**: Always open and inspect every plausible closed container at least once before ruling it out, using a checklist of unopened containers in the current room.
- **Don't**: Agent ignores or walks away from target objects that are already visible and reachable.
  **Instead**: Embed a perception buffer that immediately queues pick-up actions for visible goal objects and interrupts any ongoing exploratory loop.
- **Don't**: Agent keeps searching after acquiring enough items because it loses track of how many targets have been collected.
  **Instead**: Maintain explicit numerical goals and increment/decrement them on every successful pickup or placement, triggering transition to the next sub-goal when counts are satisfied.
- **Don't**: Agent fails to deliver a held target object to the known destination, instead continuing to wander.
  **Instead**: Adopt a hierarchical policy that, upon holding a goal object, immediately switches to a high-priority ‘deliver-to-goal’ subtask before resuming further search.
- **Don't**: Agent completes spatial placement while skipping required state changes (e.g., cleaning, heating).
  **Instead**: Integrate state precondition checks into the planner so that placement actions are only considered valid after verifying and, if necessary, executing the requisite state-changing operations.
- **Don't**: Agent repeatedly toggles or examines objects that yield no new information or progress.
  **Instead**: Implement novelty or information gain metrics that discourage repeating identical actions unless the environment state has changed since the last attempt.
- **Don't**: Agent mishandles spatial relations (e.g., ‘under’, ‘inside’, ‘behind’) by inspecting only obvious surfaces and ignoring less visible sub-locations.
  **Instead**: During task parsing, generate explicit search targets for all sub-locations associated with the reference object (drawers, shelves, underside) and iterate through them systematically.
- **Don't**: Agent forgets which object it is holding, leading to unnecessary pickups, drops, or misplacements.
  **Instead**: Maintain a dedicated ‘hand inventory’ structure updated after every pickup/drop, and consult it before issuing navigation or interaction commands.
- **Don't**: Agent executes inefficient action sequences by searching for each required object serially rather than batching placements or leveraging proximity.
  **Instead**: Use planning algorithms that group similar subtasks, allowing the agent to collect all required objects in one pass before performing consolidated delivery or transformation steps.
- **Don't**: Agent prematurely terminates or stalls without having observed any candidate objects that satisfy the goal.
  **Instead**: Only allow ‘done’ actions when explicit goal predicates are met; otherwise, continue targeted exploration of remaining unvisited rooms and containers.

성공률

해석진한 점은 seed 3개의 평균이고, 작은 빈 원은 seed별 값입니다. 세로 점선은 그 모델의 SkillRL 원래 형식 성공률입니다. RL 모델은 WikiSkill 두 방식에서 74.5~82.8 %로 원래(79.2 %)와 비슷합니다. EvoSkill 두 방식에서는 30.7~37.5 %로 떨어집니다. 원본·SFT 모델은 모든 방식이 18.2 % 이하라서 방식 사이의 차이를 읽기 어렵습니다.

• RL 모델은 SkillRL 원래 형식에서 79.2 %입니다. EvoSkill 방식에서는 30.7~37.5 %로 떨어집니다.
• 원본 모델과 SFT 모델은 모든 방식에서 18.2 % 이하입니다. 그래서 이 두 모델로는 방식끼리 비교하지 않습니다.
• 스킬을 빼면 원본 모델은 17.7 %, RL 모델은 63.0 %입니다.

재사용률

해석막대가 길수록 run 전체에서 캐시를 많이 썼습니다. 모델 세 개 모두 순서가 같습니다: WikiSkill·앞 > WikiSkill·뒤 > SkillRL 원래 > EvoSkill·공통 위 > EvoSkill.

• SkillRL 원래 형식과 WikiSkill 방식은 모델 세 개에서 0.63~0.86입니다.
• EvoSkill 방식은 0.20~0.61입니다. 모델마다 다섯 방식 중 가장 낮습니다.

한 스텝의 토큰 합과 캐시 용량

한 스텝에서 게임 64개가 보낸 프롬프트 토큰과 응답 토큰을 모두 더했습니다. 이 값은 그 스텝에 캐시가 담아야 하는 양의 근사값입니다. 빨간 선은 캐시 용량입니다.

해석선 하나가 스킬을 넣는 방식 하나입니다. 세로축은 한 스텝에서 진행 중인 게임 전체의 프롬프트·응답 토큰을 더한 값이고, 빨간 가로선은 KV 캐시 용량(41.9만 토큰)입니다. SkillRL 원래와 WikiSkill·뒤는 step 2에서 가장 크고 그 뒤로 줄어듭니다. 끝난 게임은 더 요청하지 않기 때문입니다. EvoSkill 두 방식은 대화가 쌓여서 스텝마다 커집니다. 용량선을 처음 넘는 스텝은 EvoSkill / EvoSkill·공통 위 순서로 RL 모델 22 / 27, 원본 모델 16 / 16, SFT 모델 17 / 16입니다. 위의 모델 버튼으로 바꿔 볼 수 있습니다.

해석세로축은 그 스텝 요청들의 재사용률입니다. SkillRL 원래(약 0.66)와 WikiSkill·뒤(약 0.85)는 스텝이 지나도 거의 평평합니다. EvoSkill 두 방식은 초반에 0.93~0.97로 가장 높습니다. 그런데 위 그래프에서 토큰 합이 용량선을 넘은 뒤에 크게 떨어집니다. 예를 들어 RL 모델의 EvoSkill은 step 20에서 0.97, step 30에서 0.23입니다. 두 일이 같은 시기에 일어난다는 것만 보입니다. 원인은 아직 확인하지 않았습니다.

• EvoSkill 방식은 이전 메시지를 지우지 않습니다. 그래서 스텝이 갈수록 토큰 합이 커지고, step 47~49에서 가장 큽니다.
• 조건별 최댓값은 71.5만~124.4만 토큰입니다. 캐시 용량 41.9만 토큰보다 큽니다.
• RL 모델 EvoSkill의 스텝별 재사용률은 step 20에서 0.97이고 step 30에서 0.23입니다.
• 원본 모델과 SFT 모델의 EvoSkill run에서는 선점이 run당 116~206번 있었습니다. RL 모델의 EvoSkill run은 0~3번이었습니다.

다시 계산한 토큰

해석막대가 길수록 run 하나에서 GPU가 새로 계산한 프롬프트 토큰이 많습니다. EvoSkill 두 방식은 7.9M~26.1M 토큰으로, 다른 방식(0.6M~1.5M)의 5배 이상입니다.

2-4 앞부분이 아닌 곳의 재사용

앞부분 재사용은 프롬프트 앞쪽이 같을 때만 됩니다. 스킬 구간이 프롬프트 가운데에 있으면 쓸 수 없습니다. 그래서 다른 곳에서 계산한 스킬 KV를 가져다 붙이면 무엇이 달라지는지 쟀습니다. 이런 재사용은 CacheBlend 같은 기법이 씁니다.

1. 원래 형식 프롬프트에서 구간 다섯 개를 고릅니다. 이 구간을 청크라고 부릅니다.
2. 청크의 KV를 원래 자리에서 새로 계산합니다. 이 값이 기준입니다.
3. 같은 청크의 KV를 다른 문맥에서 계산합니다. 위치 정보(RoPE)는 원래 자리에 맞게 돌립니다.
4. 두 KV의 차이를 잽니다 (K rel). 그리고 그 KV로 응답을 다시 채점합니다.

K rel

재사용한 K가 원래 K와 다른 정도를 원래 K의 크기로 나눈 값입니다. 0.15는 오차가 원래 크기의 15 %라는 뜻입니다. 층 28개와 KV 헤드 4개의 평균을 구하고, 표본 33개의 중앙값을 씁니다.

행동이 크게 바뀐 표본

각 모델이 대상 프롬프트에서 직접 뽑은 응답(temperature 0.4)을 그대로 두고, 행동 부분의 log 확률 합이 0.5보다 많이 바뀐 표본 수입니다. <action>…</action> 형식을 지키지 않은 응답은 틀린 출력으로 보고 세지 않습니다. 모델이 실제로 다른 행동을 골랐는지는 재지 않았습니다.

청크 위치와 재사용 문맥

이 실험의 대상 프롬프트는 SkillRL 원래 형식의 요청입니다. 표본은 RL 모델 run의 step 2 이후 요청 33개입니다. 아래 지도는 1-1절 예시와 같은 요청입니다.

왼쪽 글자 상자가 청크입니다. 청크는 G·C·M·H·A 다섯 개입니다. "시작"은 프롬프트에서 구간 첫 토큰의 위치입니다. chat template 토큰은 지도에서 뺐습니다. 위치에는 chat template 토큰도 셉니다.

causal attention에서는 청크 앞 토큰만 청크의 KV에 영향을 줍니다. 그래서 아래 그림에는 청크 앞 구간만 그렸습니다. 재사용 문맥에서 계산한 KV는 RoPE를 대상 위치로 다시 회전해서 씁니다.

noise 문맥은 대상 프롬프트를 청크 뒤에서 자른 문맥입니다. 수치 오차의 기준이라서 차트에서 뺐습니다.

모델별 결과

프롬프트를 어느 모델의 궤적에서 뽑았는지에 따라 두 가지로 쟀습니다. 두 경우 모두 응답은 각 모델이 직접 뽑았습니다.

• ① RL 궤적: 세 모델 모두 RL 모델이 플레이한 run의 프롬프트 33개를 씁니다. 원본·SFT 모델에게는 자기가 만들지 않은 상황입니다.
• ② 자기 궤적: 각 모델이 직접 플레이한 run(SkillRL 원래 형식, seed 0)의 프롬프트 33개를 씁니다. 뽑는 규칙은 ①과 같습니다. RL 모델은 ①과 ②가 같은 값입니다.

해석 · K rel막대가 짧을수록 다른 곳에서 계산한 KV가 원래 자리의 KV와 비슷합니다. ①에서 앞 문맥이 조금만 다른 경우(other_*, A · prev_step)는 0.02~0.13으로 짧습니다. 청크만 따로 계산한 경우(alone_*)는 0.15~0.42로 깁니다. 같은 청크·문맥에서 세 모델의 차이는 0.05 이하입니다. ②에서도 스킬 청크는 ①과 거의 같습니다. 이력 첫 항목(H)은 원본·SFT 모델에서 0.40~0.43으로 조금 깁니다. 이 두 모델의 궤적에서는 H 청크가 24~29토큰으로 RL 궤적(40토큰)보다 짧습니다.

표본 33개에서 각 모델이 뽑은 응답의 행동 형식입니다. 형식을 지키지 않은 응답은 틀린 출력입니다. 원본 모델은 ①에서 30개, ②에서 32개가 틀린 출력이라서 차트의 원본 막대는 ①에서 3개, ②에서 1개 응답만 셉니다.

해석 · 크게 바뀐 표본막대는 형식을 지킨 응답 중 행동 확률이 크게 바뀐 표본 수입니다. 0에 가까울수록 보정 없이 재사용해도 행동이 거의 그대로입니다. 스킬 청크(G·C·M)에서 RL 모델은 0~1개, SFT 모델은 0~4개입니다. 원본 모델은 3개 중 0개입니다. 행동 목록(A)을 따로 계산하면 RL 모델 12개, SFT 모델 16개로 많습니다. 원본 모델은 표본이 3개라서 이 차트로 다른 두 모델과 비교하지 못합니다.

해석 · ② 자기 궤적SFT 모델의 스킬 청크(G·C·M) 값은 ①의 0~4개에서 ②의 0~2개가 됩니다. 이력 첫 항목(H)은 SFT 모델이 ①보다 많습니다 (alone_sys 4 → 9개, prev_step 1 → 6개). 행동 목록(A) · alone_sys는 SFT 모델이 16개에서 10개로 줄어 RL 모델(12개)과 비슷합니다. A · prev_step은 짝이 되는 표본이 RL 10개, SFT 29개로 다릅니다. 원본 모델은 형식을 지킨 응답이 1개뿐이라 비교하지 못합니다. RL과 SFT 모델의 차이를 칸마다 Fisher 정확 검정으로 보면, p < 0.05인 칸은 H · alone_sys(2 vs 9, p = 0.044) 하나입니다. 13칸을 비교한 보정은 하지 않았습니다.

• 앞 문맥이 조금만 다른 경우(other_*, A · prev_step)는 K rel이 0.02~0.13입니다.
• 청크를 따로 계산한 경우(alone_*)는 K rel이 0.15~0.42입니다.
• 스킬 청크는 따로 계산해 붙여도 행동이 크게 바뀐 표본이 RL 모델 0~1개, SFT 모델 0~4개입니다 (33개 중).
• 행동 목록(A)을 따로 계산해 붙이면 33개 중 RL 모델 12개, SFT 모델 16개가 크게 바뀝니다.
• 원본 모델은 응답 33개 중 30개가 행동 형식을 지키지 않았습니다. 틀린 출력이라서 행동 변화를 세지 않았습니다.
• 각 모델은 자기가 뽑은 응답으로 채점했습니다. 이전 판은 세 모델 모두 RL 모델의 응답으로 채점해서, 원본 모델의 스킬 청크 값이 1~13개로 나왔습니다.
• ② 자기 궤적에서 SFT 모델의 스킬 청크 값은 0~2개, 행동 목록(A) · alone_sys는 10개입니다. RL 모델은 0~1개, 12개입니다.

2-5 한계

• 환경은 ALFWorld 하나입니다. WebShop과 OfficeQA는 아직 돌리지 않았습니다.
• WikiSkill과 EvoSkill 방식의 스킬 문서는 합성 문서입니다. 실제 EvoSkill 스킬은 3K~23K 문자로 더 큽니다.
• seed가 3개라서 8~10 %p보다 작은 성공률 차이는 가려내지 못합니다.
• 잰 값은 토큰 수뿐입니다. 응답 시간과 처리량은 재지 않았습니다.
• vLLM 0.11.0 한 버전, 게임 64개를 같은 걸음으로 돌린 조건입니다. 실제 서비스처럼 요청이 제각각 들어오는 조건이 아닙니다.

3 연구 질문에 대한 고민

4 skill agent 3년 계획

KV 재사용과 별개로, skill을 쓰는 에이전트로 3년 동안 무엇을 할 수 있을지 같이 브레인스토밍하려 합니다. 아래 주제들은 출발점일 뿐입니다. 아직 고른 것은 없습니다.

같이 정하고 싶은 것: 어떤 주제가 반증 가능한 한 문장으로 써지나? 1년차에 GPU 1장으로 첫 결과를 낼 수 있는 주제는 무엇인가?