Modify Uboot to Enable Shell

Note

The fastest method is to use the ch341a to dump the flash, modify it, and then reflash the modified .bin

Note

Trying to modify uboot from within initramfs or a booted openwrt does not work because the mtd2 partition apparently cannot be unlocked.

Enable uboot shell using python script

The prerequisite here is to have a stock firmware dump of the AP440; i.e., have used the ch341a to dump the flash.

Python Script

#!/usr/bin/env python3
"""
AP440 U-Boot Shell Enable Patch Script

This script patches a 4MB SPI flash dump to enable the U-Boot shell.
The shell control flag is located at offset 0x3a0003 in the datto partition.

Usage:
    python3 enable_shell_patch.py <input_file> <output_file>

Example:
    python3 enable_shell_patch.py stock.bin stock-shell-enabled.bin
"""

import sys
import os

# Shell control flag location
SHELL_FLAG_OFFSET = 0x3a0003
SHELL_DISABLED = 0x01
SHELL_ENABLED = 0x00

def patch_shell(input_file, output_file):
    """Patch the firmware to enable U-Boot shell"""

    # Read the input firmware
    print(f"Reading {input_file}...")
    with open(input_file, "rb") as f:
        firmware = bytearray(f.read())

    if len(firmware) != 0x400000:  # 4MB
        print(f"WARNING: Expected 4MB (4194304 bytes), got {len(firmware)} bytes")

    # Check current shell flag
    original_byte = firmware[SHELL_FLAG_OFFSET]
    print(f"\nCurrent shell flag at 0x{SHELL_FLAG_OFFSET:06x}: 0x{original_byte:02x} ", end="")
    print(f"({'DISABLED' if original_byte == SHELL_DISABLED else 'ENABLED'})")

    if original_byte == SHELL_ENABLED:
        print("Shell is already enabled!")
        response = input("Continue anyway? (y/n): ")
        if response.lower() != 'y':
            print("Aborted.")
            return False

    # Patch to enable shell
    firmware[SHELL_FLAG_OFFSET] = SHELL_ENABLED
    print(f"Modified shell flag at 0x{SHELL_FLAG_OFFSET:06x}: 0x{SHELL_ENABLED:02x} (ENABLED)")

    # Write modified firmware
    print(f"\nWriting {output_file}...")
    with open(output_file, 'wb') as f:
        f.write(firmware)

    print(f"Created {output_file} ({len(firmware)} bytes)")

    # Verify the change
    with open(output_file, 'rb') as f:
        f.seek(0x3a0000)
        shell_control = f.read(16)

    print(f"\nVerification - Shell control structure at 0x3a0000:")
    print(" ".join(f"{b:02x}" for b in shell_control))

    if shell_control[3] == SHELL_ENABLED:
        print("\n✓ Patch successful! Ready to flash with programmer.")
        return True
    else:
        print("\n✗ Verification failed!")
        return False

def main():
    if len(sys.argv) != 3:
        print(__doc__)
        sys.exit(1)

    input_file = sys.argv[1]
    output_file = sys.argv[2]

    if not os.path.exists(input_file):
        print(f"Error: Input file '{input_file}' not found")
        sys.exit(1)

    if os.path.exists(output_file):
        response = input(f"Output file '{output_file}' exists. Overwrite? (y/n): ")
        if response.lower() != 'y':
            print("Aborted.")
            sys.exit(0)

    success = patch_shell(input_file, output_file)
    sys.exit(0 if success else 1)

if __name__ == "__main__":
    main()

Enable the uboot shell from within uboot

Note

You can perform these steps manually if you are already in a uboot shell or use the bootcmd: set it by using the fwupgrade.cfg + glitch or if you are already in openwrt (e.g. initramfs) use the fw_setenv bootcmd '' and the command [[Boot Commands#uboot shell fix and bootcmd change| here]]

Manual instructions (if you are already in uboot somehow)

Read datto partition to RAM

sf probe
sf read 0x42000000 0x3a0000 0x8000

Check current value

md.b 0x42000000 0x10

Patch byte 3 (change 0x01 to 0x00)

mw.b 0x42000003 0x00 1

Verify

md.b 0x42000000 0x10

Erase and write back

sf erase 0x3a0000 0x10000
sf write 0x42000000 0x3a0000 0x8000

Verify write

sf read 0x43000000 0x3a0000 0x10
md.b 0x43000000 0x10

Random notes

Prerequisite: be able to get back to the uboot shell.