GUI and Graphics Development¶

This guide explains how to develop graphical applications using the VNC/noVNC setup in the ESP32 Template development container.

Overview¶

The dev container includes a complete desktop environment for GUI development:

Fluxbox Window Manager: Lightweight desktop environment
TigerVNC Server: VNC server running on display :1
noVNC Web Client: Browser-based VNC access
X11 Support: Full X Window System for GUI applications

This setup allows you to:

Run GUI applications from the terminal (they appear on the desktop)
Access the desktop via web browser at port 6080
Use VNC clients for better performance
Develop and test graphical tools without leaving Codespaces

Architecture¶

Desktop Stack¶

┌─────────────────────────────────────────────┐
│  Browser (noVNC) or VNC Client              │
└──────────────────┬──────────────────────────┘
                   │ Port 6080 (HTTP+WebSocket)
                   │ Port 5901 (VNC Protocol)
┌──────────────────▼──────────────────────────┐
│  noVNC Server (websockify)                  │
│  - HTTP server for web interface            │
│  - WebSocket → VNC protocol bridge          │
└──────────────────┬──────────────────────────┘
                   │ VNC Protocol
┌──────────────────▼──────────────────────────┐
│  TigerVNC Server (:1)                       │
│  - DISPLAY=:1                               │
│  - Resolution: 1024x768 @ 24-bit            │
│  - TMPDIR=/home/esp/vnc-tmp                 │
└──────────────────┬──────────────────────────┘
                   │ X11 Protocol
┌──────────────────▼──────────────────────────┐
│  Fluxbox Window Manager                     │
│  - Lightweight, minimal resource usage      │
│  - Right-click for application menu         │
└──────────────────┬──────────────────────────┘
                   │
┌──────────────────▼──────────────────────────┐
│  X11 Applications                           │
│  - GUI tools (gtkwave, etc.)                │
│  - VS Code terminal commands                │
│  - Custom graphical applications            │
└─────────────────────────────────────────────┘

Accessing the Desktop¶

Via Web Browser (Recommended for Codespaces)¶

Automatic Port Forwarding: Codespaces automatically forwards port 6080
Open in Browser: Click on “Ports” tab → Port 6080 → Click globe icon
Full Screen: Use browser’s full screen mode (F11) for better experience

Note

noVNC is perfect for Codespaces as it requires no client installation and works in any browser.

Via VNC Client (Better Performance)¶

For local Docker containers or better performance:

# Find your container's IP or use port forwarding
# Connect VNC client to: localhost:5901
# Password: (typically 'vscode' or none)

Recommended VNC Clients:

RealVNC Viewer (cross-platform)
TigerVNC Viewer (Linux)
TightVNC (Windows)

VNC Setup Details¶

Automatic Startup¶

VNC server starts automatically when the dev container launches:

Startup Script: .devcontainer/setup-vnc.sh
systemd Integration: VNC runs as a service
TMPDIR Workaround: Uses /home/esp/vnc-tmp instead of /tmp

# Check VNC status
ps aux | grep vnc

# View VNC log
tail -f /home/esp/.vnc/*.log

# Restart VNC if needed
vncserver -kill :1
vncserver :1 -geometry 1024x768 -depth 24

Key Configuration Files¶

``.devcontainer/setup-vnc.sh``:

#!/bin/bash
# VNC server startup script

export TMPDIR=/home/esp/vnc-tmp
mkdir -p $TMPDIR

vncserver :1 \
  -geometry 1024x768 \
  -depth 24 \
  -localhost no \
  -SecurityTypes None

``.vnc/xstartup``:

#!/bin/bash
unset SESSION_MANAGER
unset DBUS_SESSION_BUS_ADDRESS

export TMPDIR=/home/esp/vnc-tmp

[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources

xsetroot -solid grey
exec fluxbox

TMPDIR Workaround¶

Problem: Codespaces mounts /tmp as read-only, causing VNC to fail.

Solution: Use custom TMPDIR for VNC temporary files.

# Environment variable set in all VNC-related scripts
export TMPDIR=/home/esp/vnc-tmp

# Directory is writable by container user
mkdir -p /home/esp/vnc-tmp
chmod 700 /home/esp/vnc-tmp

Running GUI Applications¶

From VS Code Terminal¶

Any GUI application launched from the terminal appears on the VNC desktop:

# Set display (usually automatic)
export DISPLAY=:1

# Launch GUI application
firefox &

# Launch VS Code (if available)
code . &

# Run custom GUI tool
./my_gui_app &

Common GUI Tools in Container:

xterm: Terminal emulator
xcalc: Calculator (for testing)
File manager (if installed)
Custom debugging tools

Example: GTKWave for Waveform Viewing¶

# Install GTKWave (if needed)
sudo apt-get update && sudo apt-get install -y gtkwave

# Launch GTKWave
gtkwave waveform.vcd &

# Application window appears on VNC desktop

Developing Custom GUI Applications¶

ESP32 GUI Frameworks¶

While the VNC desktop is for host-side tools, ESP32 can drive real displays:

For ESP32 Display Output:

LVGL (Light and Versatile Graphics Library)
ESP-IDF LCD/Display Drivers
SPI/I2C Display Libraries

Host Development Tools:

Use VNC desktop for testing GUI tool scripts
Visualize sensor data with Python matplotlib
Debug display layouts before deploying to ESP32

Python GUI Example¶

# gui_test.py - Simple GUI application
import tkinter as tk

root = tk.Tk()
root.title("ESP32 Configuration Tool")

label = tk.Label(root, text="Hello from VNC Desktop!")
label.pack(pady=20)

button = tk.Button(root, text="Close", command=root.quit)
button.pack()

root.mainloop()

# Run the GUI app (appears on VNC desktop)
python3 gui_test.py

C++ GUI with GTK (Advanced)¶

// example_gtk.cpp
#include <gtk/gtk.h>

static void on_button_clicked(GtkWidget *widget, gpointer data) {
    g_print("Button clicked!\n");
}

int main(int argc, char *argv[]) {
    gtk_init(&argc, &argv);

    GtkWidget *window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    gtk_window_set_title(GTK_WINDOW(window), "ESP32 Tool");

    GtkWidget *button = gtk_button_new_with_label("Click Me");
    g_signal_connect(button, "clicked",
                    G_CALLBACK(on_button_clicked), NULL);

    gtk_container_add(GTK_CONTAINER(window), button);
    gtk_widget_show_all(window);

    gtk_main();
    return 0;
}

# Compile and run
g++ example_gtk.cpp -o example_gtk `pkg-config --cflags --libs gtk+-3.0`
./example_gtk &

Troubleshooting¶

VNC Server Not Starting¶

Symptom: Cannot connect to VNC on port 6080

# Check if VNC server is running
ps aux | grep vnc

# Check VNC log for errors
cat /home/esp/.vnc/*.log

# Common issue: TMPDIR not writable
ls -ld /home/esp/vnc-tmp

# Restart VNC with correct TMPDIR
export TMPDIR=/home/esp/vnc-tmp
vncserver -kill :1
vncserver :1 -geometry 1024x768 -depth 24

Cannot See GUI Applications¶

Symptom: Applications launch but don’t appear on desktop

# Verify DISPLAY is set correctly
echo $DISPLAY  # Should show :1

# Set DISPLAY if needed
export DISPLAY=:1

# Test with simple GUI
xterm &

# Check X server errors
xdpyinfo -display :1

Black Screen in noVNC¶

Symptom: VNC connects but shows black screen

# Fluxbox might not be running
ps aux | grep fluxbox

# Restart Fluxbox
DISPLAY=:1 fluxbox &

# Or restart entire VNC session
vncserver -kill :1
export TMPDIR=/home/esp/vnc-tmp
vncserver :1 -geometry 1024x768 -depth 24

Performance Issues¶

Symptom: Slow or laggy VNC connection

Use native VNC client instead of noVNC (better performance)
Reduce color depth: Change -depth 24 to -depth 16
Lower resolution: Use 1024x768 instead of higher resolutions
Close unused applications on the VNC desktop

Port 6080 Already in Use¶

# Find what's using the port
sudo lsof -i :6080

# Kill the process
sudo kill -9 <PID>

# Restart noVNC
websockify --web=/usr/share/novnc 6080 localhost:5901 &

Best Practices¶

Development Workflow¶

Use Terminal First: Most ESP-IDF work is command-line based
GUI for Visualization: Use VNC desktop for:
- Waveform viewing (GTKWave)
- Data plotting (Python matplotlib)
- Custom debugging tools
- Web browser testing
Keep Desktop Clean: Close unused GUI apps to save resources
Persistent Sessions: VNC survives terminal disconnections

Resource Management¶

# Monitor container resource usage
top

# Check memory usage
free -h

# Kill heavy GUI processes if needed
pkill firefox
pkill chrome

Security Considerations¶

VNC Security in Codespaces:

VNC runs without password (container is private)
Codespaces port forwarding is authenticated via GitHub
Do NOT expose port 5901 publicly without authentication

For Production Use:

# Set VNC password
vncpasswd

# Start with security
vncserver :1 -SecurityTypes VncAuth

Use Cases¶

Common Development Scenarios¶

1. Testing Web Interfaces

# Build and run ESP32 web server in QEMU
idf.py build
./tools/run-qemu-network.sh

# Open browser on VNC desktop
firefox http://10.0.2.15 &

2. Visualizing Sensor Data

# plot_sensor.py
import matplotlib.pyplot as plt
import serial

# Read from ESP32 serial port
data = []
with serial.Serial('/dev/ttyUSB0', 115200) as ser:
    for _ in range(100):
        data.append(float(ser.readline()))

# Plot on VNC desktop
plt.plot(data)
plt.show()

3. Debugging with gtkwave

# Generate VCD file from logic analyzer
idf.py build

# View waveforms
gtkwave debug/signals.vcd &

Integration with VS Code¶

Opening Files from Desktop¶

# From VNC terminal (xterm)
code /workspaces/esp32-template/main/main.c

# File opens in VS Code editor

Running VS Code Tasks¶

GUI applications can be launched from VS Code tasks:

// .vscode/tasks.json
{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "View Waveform",
      "type": "shell",
      "command": "DISPLAY=:1 gtkwave ${file} &",
      "problemMatcher": []
    }
  ]
}

Advanced Configuration¶

Custom Desktop Resolution¶

# Stop current VNC server
vncserver -kill :1

# Start with custom resolution
export TMPDIR=/home/esp/vnc-tmp
vncserver :1 -geometry 1920x1080 -depth 24

Multiple VNC Displays¶

# Start second display
vncserver :2 -geometry 800x600 -depth 24

# Applications on different displays
DISPLAY=:1 firefox &
DISPLAY=:2 gtkwave waveform.vcd &

Alternative Window Managers¶

Replace Fluxbox with other window managers:

# Install alternative WM
sudo apt-get install -y openbox

# Edit ~/.vnc/xstartup
# Replace 'exec fluxbox' with 'exec openbox'

GUI and Graphics Development¶

Overview¶

Architecture¶

Desktop Stack¶

Accessing the Desktop¶

Via Web Browser (Recommended for Codespaces)¶

Via VNC Client (Better Performance)¶

VNC Setup Details¶

Automatic Startup¶

Key Configuration Files¶

TMPDIR Workaround¶

Running GUI Applications¶

From VS Code Terminal¶

Example: GTKWave for Waveform Viewing¶

Developing Custom GUI Applications¶

ESP32 GUI Frameworks¶

Python GUI Example¶

C++ GUI with GTK (Advanced)¶

Troubleshooting¶

VNC Server Not Starting¶

Cannot See GUI Applications¶

Black Screen in noVNC¶

Performance Issues¶

Port 6080 Already in Use¶

Best Practices¶

Development Workflow¶

Resource Management¶

Security Considerations¶

Use Cases¶

Common Development Scenarios¶

Integration with VS Code¶

Opening Files from Desktop¶

Running VS Code Tasks¶

Advanced Configuration¶

Custom Desktop Resolution¶

Multiple VNC Displays¶

Alternative Window Managers¶

References¶

See Also¶