Reference ========= Game description fields ----------------------- Game descriptions are written in the ``[tool.retro-gamer]`` section of your game project's ``pyproject.toml``. ``retro-gamer init`` reads this section and copies the metadata into the training run's ``config.toml``, where it can also be inspected or hand-edited. A complete example for the Snake game: .. code-block:: toml [tool.retro-gamer] factory = "snake:create_game" actions = ["KEY_RIGHT", "KEY_UP", "KEY_LEFT", "KEY_DOWN"] reward = "score" character_set = ["@", "*", ">", "<", "^", "v"] You do not need to specify the board size: ``retro-gamer`` reads it directly from your game's ``board_size`` attribute. The fields are described below. ``factory`` ~~~~~~~~~~~ **Required.** A ``"module:attr"`` string naming the function that creates a fresh game instance. The function must take no arguments and return a new :class:`retro.game.Game`. .. code-block:: toml factory = "snake:create_game" ``actions`` ~~~~~~~~~~~ **Required.** A list of keystroke names the agent may send to the game each turn. Use arrow key names for directional games, or single characters for character-key games. .. code-block:: toml actions = ["KEY_RIGHT", "KEY_UP", "KEY_LEFT", "KEY_DOWN"] The agent also has access to a no-op action (doing nothing). The total number of actions in the Q-network output is ``len(actions) + 1``. ``reward`` ~~~~~~~~~~ **Required.** The key in the game's state dictionary to use as the reward signal. The reward computed for each turn is the *change* in this value from the previous turn. .. code-block:: toml reward = "score" ``character_set`` ~~~~~~~~~~~~~~~~~ **Optional.** A list of single characters that may appear on the board. Each character occupies one "slot" in the one-hot encoding. Characters not in this list are treated as empty space. .. code-block:: toml character_set = ["@", "*", ">", "<", "^", "v"] If omitted, ``retro-gamer`` runs an exploration phase to discover the characters that appear in practice. The length of this phase is controlled by the ``exploration_turns`` hyperparameter. Preprocessing options --------------------- Preprocessing options live in the ``[preprocessing]`` section of a run's ``config.toml``. They control how the game's board and state are transformed into the observation vector that the neural network sees. ``retro-gamer init`` writes sensible defaults; you can edit them by hand before running ``retro-gamer train``. .. note:: Changes to any ``[preprocessing]`` option—or to the game description fields above—make existing checkpoints incompatible. Run ``retro-gamer clean`` before retraining after such changes. ``spatial`` (default: ``false``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Whether to treat the board as a 2D spatial scene. When ``true``, the trainer uses a convolutional neural network (CNN); when ``false``, a multilayer perceptron (MLP) that sees the board as a flat list of numbers. ``board`` (default: ``true``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Whether to include the board encoding in the observation vector. Set to ``false`` to train on game state variables only, with no board at all. This is useful for games with small, enumerable state spaces where a lookup table (classic Q-learning) is sufficient. When ``board = false``: - ``spatial`` must also be ``false`` (no board means no 2D scene for a CNN). - At least one key must be listed in ``observe_state``. - ``character_set`` is not required and character discovery is skipped. .. code-block:: toml [preprocessing] board = false observe_state = ["board_state"] ``observe_state`` (default: ``[]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A list of keys from ``game.state`` to include in the observation vector, appended after the board encoding (or as the entire observation when ``board = false``). Scalar values contribute one element each; list or tuple values are flattened. .. code-block:: toml observe_state = ["apple_dx", "apple_dy"] The keys must be present in ``game.state`` at every step, initialized before the game starts. All values that are lists or tuples must always have the same length from episode to episode. .. warning:: ``observe_state`` keys must be initialized to their final shape before the game starts. If a key is absent or its list length changes between episodes, training will crash with an error explaining which key changed and by how much. This happens because the neural network's input layer has a fixed size determined at the start of training; it cannot adapt to a changing observation shape mid-run. Always initialize every observed key with a placeholder of the correct type and length before the first ``game.step()`` call. .. _observation-function: ``observation_function`` (default: none) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ **Optional**, set in ``[metadata]`` (alongside ``actions``/``reward``/ ``character_set``/``board_size``), not in ``[preprocessing]``. A ``"module:attr"`` string naming a function ``f(game) -> observation`` that fully replaces the built-in board/``observe_state`` encoding described above. Mutually exclusive with ``observe_state`` — they're two conflicting ways of describing the same thing, and setting both raises an error. .. code-block:: toml [metadata] observation_function = "my_game:get_observation" For DQN training, the function must return a flat, numeric, fixed-length 1-D array every time it's called — the same contract the built-in encoder follows: a flattened one-hot board (sized from ``character_set`` × ``board_size``, if ``board = true``) followed by any extra features, all in one vector. ``character_set`` and ``board_size`` stay required either way, because that's what lets ``observation_function`` also use a spatial (``spatial = true``) network — the trainer slices the flat vector back into a board tensor using exactly those two fields, the same way it does for the built-in encoder. ``observation_function`` is the training-side complement to the game's state dict. Use ``observe_state`` when the game already computes the features you want and stores them in ``game.state``; use ``observation_function`` when you want to transform the game's board or state into a representation that is more useful for learning — for example, mapping all obstacle types to a single character, or cropping the board to an egocentric window centred on the player. This is how you get an egocentric (cropped, player-centered) board — call ``egocentric_board()`` and ``encode_board()`` yourself, from :mod:`retro_gamer.observation`, inside your own function, and declare ``board_size`` to match your crop: .. code-block:: python import numpy as np from retro.views.headless import HeadlessView from retro_gamer.observation import egocentric_board, encode_board, encode_state CHARACTER_SET = ["@", "*", ">", "<", "^", "v"] RADIUS = 8 def egocentric_observation(game): view = HeadlessView() view.on_game_start(game) view.render(game) head = game.get_agent_by_name("Snake head") cropped = egocentric_board(view.board_characters, head.position, RADIUS) board_vec = encode_board(cropped, CHARACTER_SET).flatten() extras = encode_state(game.state, ["apple_dx", "apple_dy"]) return np.concatenate([board_vec, extras]) .. code-block:: toml [metadata] board_size = [17, 17] # 2*RADIUS + 1 observation_function = "my_module:egocentric_observation" Outside DQN training — for example, a tabular Q-learning lab that uses :class:`~retro_gamer.GameEnvironment` directly without :class:`~retro_gamer.DQNTrainer` — there's no 1-D requirement at all. ``observation_function`` can return anything you want to use as your observation, including a plain tuple used as a dict key. .. _hyperparameters: Hyperparameters --------------- Hyperparameters are split across two sections of ``config.toml``: - ``[model]`` — network architecture (changing these requires starting fresh) - ``[training]`` — learning algorithm parameters (safe to change at any time) Both sections can be set via ``retro-gamer init`` options or edited directly in ``config.toml`` between ``init`` and ``train``. Learning and optimization (``[training]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``learning_rate`` (default: ``0.0001``) The step size used by the Adam optimizer when updating network weights. Larger values converge faster but may be unstable; smaller values are more stable but slower. ``learning_rate_decay`` (default: ``0.9999``) Multiplicative decay applied to the learning rate after each episode. The learning rate decreases geometrically over training, helping the network fine-tune later without destabilizing early progress. With the default value, the learning rate decays to about 13 % of its starting value after 20 000 episodes. ``gamma`` (default: ``0.99``) The discount factor for future rewards. A value of 1.0 makes the agent value all future rewards equally; smaller values make the agent increasingly myopic. Exploration (``[training]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``epsilon`` (default: ``1.0``) The initial exploration rate. At each turn, the agent takes a random action with probability ``epsilon`` and exploits its current Q-function with probability ``1 - epsilon``. ``epsilon_decay`` (default: ``0.9997``) Multiplicative decay applied to ``epsilon`` after each episode. ``epsilon_min`` (default: ``0.05``) The floor below which ``epsilon`` will not fall. A small amount of continued exploration prevents the agent from becoming permanently committed to a suboptimal policy. Memory and sampling (``[training]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``batch_size`` (default: ``64``) The number of experiences sampled from the replay buffer per training step. ``memory_capacity`` (default: ``50000``) The maximum number of experiences the replay buffer can hold. When full, the oldest experiences are discarded. ``prioritize_experiences`` (default: ``true``) Whether to use prioritized experience replay. When ``true``, experiences with larger TD errors are sampled more frequently. This often improves sample efficiency at a modest computational cost. Model architecture (``[model]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Changing any ``[model]`` option requires starting fresh (run ``retro-gamer clean`` before retraining). ``hidden_sizes`` (default: ``[128, 64]``) A list of integers giving the size of each hidden layer in the MLP head. The default creates two layers: 128 units then 64. For spatial games this follows the CNN; for non-spatial games it is the full network. Larger or deeper networks can represent more complex Q-functions but train more slowly and may need more episodes. Training duration (``[training]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``training_episodes`` (default: ``20000``) The total number of game episodes to run. Each episode runs until the game ends or ``max_turns_per_episode`` turns have elapsed. ``max_turns_per_episode`` (default: ``2000``) A safety cutoff preventing a single episode from running indefinitely (for example, if the agent finds a way to avoid dying). ``target_update_freq`` (default: ``500``) How many training steps between updates of the target network. More frequent updates make training targets move faster (less stable); less frequent updates make them more stable but slower to reflect new learning. ``train_every`` (default: ``4``) Run one training step every N game steps. Higher values speed up episode collection at the cost of fewer gradient updates per experience. The default of 4 is a good balance for most games; set to 1 to train on every step. Character discovery (``[training]``) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``exploration_turns`` (default: ``200``) When ``character_set`` is not specified, the number of random turns to run at the start of training to discover which characters appear on the board. ``unknown_character_strategy`` (default: ``"ignore"``) What to do when a character appears during training that is not in the established ``character_set``. ``"ignore"`` treats it as an empty cell; ``"extend"`` rebuilds the model with an extended character set. CLI reference ------------- ``retro-gamer init`` ~~~~~~~~~~~~~~~~~~~~~ Create a new training run directory with ``config.toml``. Game metadata is read automatically from the ``[tool.retro-gamer]`` section of your game's ``pyproject.toml``; you do not pass it on the command line. .. code-block:: console % retro-gamer init GAME OUTPUT [OPTIONS] **Required arguments:** - ``GAME`` — Your game, specified as a directory path or a Python module name: - Directory: ``games/snake`` - Module name: ``retro.examples.snake`` The ``[tool.retro-gamer]`` section is read from the ``pyproject.toml`` found in or above the game directory. - ``OUTPUT`` — Directory to create for this training run (e.g. ``training/snake``). **Hyperparameter options** (all optional; see :ref:`hyperparameters`): - ``--training-episodes N`` - ``--hidden-sizes SIZES`` — comma-separated, e.g. ``512,256`` - ``--learning-rate F`` - ``--learning-rate-decay F`` - ``--gamma F`` - ``--epsilon-decay F`` - ``--epsilon-min F`` - ``--batch-size N`` - ``--memory-capacity N`` - ``--target-update-freq N`` - ``--max-turns-per-episode N`` - ``--exploration-turns N`` - ``--train-every N`` - ``--prioritize-experiences`` / ``--no-prioritize-experiences`` ``retro-gamer train`` ~~~~~~~~~~~~~~~~~~~~~ Train a DQN agent. .. code-block:: console % retro-gamer train RUN_DIR ``RUN_DIR`` must contain a ``config.toml`` generated by ``retro-gamer init``. If checkpoints already exist in ``RUN_DIR``, training automatically resumes from the latest one so prior work is never lost. If all configured episodes have already been completed, the command prints a message and exits immediately. To keep training, increase ``training_episodes`` in ``config.toml`` and run again. **Incompatible changes.** Some config changes make existing checkpoints unusable. If you change any of the following, ``retro-gamer train`` will detect the mismatch and refuse to resume, with a clear explanation: - ``actions``, ``reward``, ``character_set``, ``board_size``, ``observation_function`` (``[metadata]``) — game description and observation shape - ``spatial``, ``board``, ``observe_state`` (``[preprocessing]``) — observation encoding - ``hidden_sizes`` (``[model]``) — network architecture Run ``retro-gamer clean RUN_DIR`` to remove the old checkpoints and start fresh. Other hyperparameter changes (learning rate, epsilon, etc.) are safe and take effect immediately on the next training run. ``retro-gamer play`` ~~~~~~~~~~~~~~~~~~~~ Watch a trained agent play the game in the terminal. .. code-block:: console % retro-gamer play RUN_DIR [--checkpoint NAME] [--framerate N] By default, the latest available checkpoint is loaded. Use ``--checkpoint`` to load a specific one by name (e.g. ``ep_0100``). ``--framerate`` sets the target frames per second (default: 12). Press Enter or Escape to quit. ``retro-gamer clean`` ~~~~~~~~~~~~~~~~~~~~~ Remove all checkpoints and the training log from a run directory. .. code-block:: console % retro-gamer clean RUN_DIR Prompts for confirmation before deleting. Use ``--yes`` / ``-y`` to skip the prompt. The ``config.toml`` is preserved so you can run ``retro-gamer train`` immediately to start fresh with the same settings. Use this after making an incompatible change (see ``retro-gamer train`` above) or any time you want to restart training from scratch. ``retro-gamer info`` ~~~~~~~~~~~~~~~~~~~~~ Print a summary of a training run: metadata, hyperparameters, recent checkpoint log, and available checkpoints. .. code-block:: console % retro-gamer info RUN_DIR Training run directory structure --------------------------------- A training run is a self-contained directory with the following contents: .. code-block:: text training/snake/ ├── config.toml # game description + hyperparameters ├── training.log # architecture rationale + per-episode log └── checkpoints/ ├── ep_0100.pt # model weights at episode 100 ├── ep_0200.pt └── ... # one file saved every 100 episodes ``config.toml`` is written by ``retro-gamer init`` and updated (with the discovered character set and resolved hyperparameters) when ``retro-gamer train`` begins. It has five sections: ``[game]``, ``[metadata]``, ``[preprocessing]``, ``[model]``, and ``[training]``. Editing ``config.toml`` between ``init`` and ``train`` is the recommended way to adjust hyperparameters. ``training.log`` begins with the full network architecture description, then one line per checkpoint (every 100 episodes) in the format:: [ep_NNNN] ep=SSSS-NNNN avg_reward=F avg_steps=N epsilon=F avg_loss=F time=Xm Xs total=Xm Xs Each field averages over the episodes since the previous checkpoint: - ``ep=SSSS-NNNN`` — episode range covered by this entry - ``avg_reward`` — mean total reward per episode (positive = good) - ``avg_steps`` — mean episode length in game turns - ``epsilon`` — current exploration rate (approaches ``epsilon_min`` over time) - ``avg_loss`` — mean Huber loss across training steps (should decrease as learning stabilises). Huber loss equals ½·(q−t)² for small errors and |q−t|−½ for large ones, so it stays bounded even when Q-values are large. Values in the range 0–10 are typical; a slow downward trend over thousands of episodes is the healthy pattern. A loss that grows without bound indicates a learning rate that is too high. - ``time`` — wall-clock time for this checkpoint interval - ``total`` — cumulative training time across all sessions When training is resumed, a ``=== Resumed from ... ===`` line is appended so the log records the full history of a run across multiple sessions. Python API ---------- For advanced use, ``retro-gamer``'s components are importable as a library. See the :doc:`api` reference for full details. .. code-block:: python from retro_gamer import GameMetadata, DQNTrainer from retro.examples.snake import create_game metadata = GameMetadata.from_pyproject("retro.examples.snake") trainer = DQNTrainer(create_game, metadata, "training/snake/") trainer.train()