Building Production-Grade AI Features in Flutter: Architecture, Streaming, and State

Most introductory guides on integrating AI into Flutter tell you to grab your API key and paste it directly into an http.post call inside a widget or state notifier.

In production, that approach causes immediate problems:

1. Leaked Keys: Anyone can decompile an APK or inspect an IPA using strings or jadx and extract hardcoded API keys in under a minute.
2. Unacceptable Latency: Waiting 8 to 15 seconds for a complete HTTP response while showing a spinner leads to high abandonment rates.
3. Flaky Connections: Mobile devices jump between 5G, LTE, and dead zones. If a 12-second HTTP call drops at second 11, the provider bills you for all generated tokens while the user sees a blank error screen.
4. No Cost Controls: If your client talks directly to OpenAI or Anthropic, a malicious user can intercept the request and burn your API credits.

Here is how we structure production Flutter AI features to avoid these pitfalls.

The Production Pattern: Edge Gateway Architecture

Mobile clients should never talk directly to AI providers. Instead, route requests through a thin, authenticated Edge Gateway (such as a Cloudflare Worker or simple Node backend):

+-----------------------+                +-------------------------+                +--------------------+
|  Flutter Application  |                | Cloudflare Worker / API |                |    AI Provider     |
|                       |                |   (Edge Gateway Layer)  |                |  (Anthropic/OpenAI)|
|  - UI State (Riverpod)|                |                         |                |                    |
|  - SSE Byte Decoder   |  Bearer JWT    |  - User Authentication  |   Private Key  |  - Raw Token Gen   |
|  - 60/120 FPS Render  | -------------> |  - Per-User Rate Limits | -------------> |                    |
|  - Local Cache (SQLite|                |  - Prompt Injection Wall|                |                    |
|                       | <============= |  - Token Cost Tracking  | <============= |                    |
+-----------------------+   Chunked SSE  +-------------------------+   HTTP Stream  +--------------------+

The gateway handles three critical jobs:

• Authentication & Rate Limiting: Verifies the user's session token and enforces daily request limits before hitting the model provider.
• Key Isolation: Your proprietary provider keys live only on the edge server, never in client binaries.
• Cost Protection: Restricts max output tokens and enforces prompt guardrails server-side.

Streaming Server-Sent Events (SSE) in Flutter

To give users an immediate response, stream tokens as they are generated. In Flutter, we handle SSE streams using http.Client and reactive Streams:

import 'dart:async';
import 'dart:convert';
import 'package:http/http.dart' as http;

class AiStreamingService {
  final http.Client _client;
  AiStreamingService({http.Client? client}) : _client = client ?? http.Client();

  /// Streams token deltas from your authenticated backend gateway
  Stream<String> streamAiCompletion({
    required String prompt,
    required String userJwt,
  }) async* {
    final request = http.Request(
      'POST',
      Uri.parse('https://api.yourdomain.com/v1/ai/stream'),
    );

    request.headers.addAll({
      'Content-Type': 'application/json',
      'Authorization': 'Bearer $userJwt',
      'Accept': 'text/event-stream',
    });

    request.body = jsonEncode({
      'prompt': prompt,
      'stream': true,
    });

    http.StreamedResponse response;
    try {
      response = await _client.send(request);
    } catch (e) {
      throw Exception('Network failure reaching AI gateway: $e');
    }

    if (response.statusCode != 200) {
      final errorBody = await response.stream.bytesToString();
      throw Exception('Gateway error (' + response.statusCode.toString() + '): ' + errorBody);
    }

    // Decode incoming byte chunks using utf8 and line splitters
    String buffer = '';
    await for (final chunk in response.stream.transform(utf8.decoder)) {
      buffer += chunk;
      final lines = buffer.split('\n');
      
      // Keep unfinished line fragment in buffer
      buffer = lines.removeLast();

      for (final line in lines) {
        final trimmed = line.trim();
        if (trimmed.isEmpty || trimmed.startsWith(':')) continue; // Heartbeats

        if (trimmed.startsWith('data: ')) {
          final data = trimmed.substring(6).trim();
          if (data == '[DONE]') return;

          try {
            final json = jsonDecode(data);
            final delta = json['delta'] as String?;
            if (delta != null && delta.isNotEmpty) {
              yield delta;
            }
          } catch (_) {
            // Ignore partial JSON chunks until complete line arrives
          }
        }
      }
    }
  }

  void dispose() {
    _client.close();
  }
}

Managing Stream State in the UI

Streaming text directly into a standard widget rebuild on every token can cause frame stutter if you're not careful. Use a dedicated StateNotifier or Riverpod provider to accumulate tokens into an immutable message model:

import 'package:flutter/foundation.dart';
import 'package:flutter_riverpod/flutter_riverpod.dart';

@immutable
class ChatMessage {
  final String id;
  final String text;
  final bool isUser;
  final bool isStreaming;

  const ChatMessage({
    required this.id,
    required this.text,
    required this.isUser,
    this.isStreaming = false,
  });

  ChatMessage copyWith({String? text, bool? isStreaming}) {
    return ChatMessage(
      id: id,
      text: text ?? this.text,
      isUser: isUser,
      isStreaming: isStreaming ?? this.isStreaming,
    );
  }
}

class ChatNotifier extends StateNotifier<List<ChatMessage>> {
  final AiStreamingService _aiService;
  final String _userJwt;

  ChatNotifier(this._aiService, this._userJwt) : super([]);

  Future<void> sendMessage(String text) async {
    final userMsg = ChatMessage(
      id: DateTime.now().millisecondsSinceEpoch.toString(),
      text: text,
      isUser: true,
    );

    final aiMsgId = (DateTime.now().millisecondsSinceEpoch + 1).toString();
    final placeholder = ChatMessage(
      id: aiMsgId,
      text: '',
      isUser: false,
      isStreaming: true,
    );

    // Optimistically update list with user message and streaming placeholder
    state = [...state, userMsg, placeholder];

    try {
      final stream = _aiService.streamAiCompletion(
        prompt: text,
        userJwt: _userJwt,
      );

      String accumulatedText = '';
      await for (final token in stream) {
        accumulatedText += token;
        
        // Update the last item efficiently
        state = [
          for (int i = 0; i < state.length; i++)
            if (state[i].id == aiMsgId)
              state[i].copyWith(text: accumulatedText)
            else
              state[i]
        ];
      }

      // Mark stream as complete
      state = [
        for (int i = 0; i < state.length; i++)
          if (state[i].id == aiMsgId)
            state[i].copyWith(isStreaming: false)
          else
            state[i]
      ];
    } catch (e) {
      // Handle network drop or error gracefully
      state = [
        for (int i = 0; i < state.length; i++)
          if (state[i].id == aiMsgId)
            state[i].copyWith(
              text: 'Connection dropped. Tap to retry.',
              isStreaming: false,
            )
          else
            state[i]
      ];
    }
  }
}

Handling Mobile Failure Modes

In mobile production, network failures are the norm, not the exception:

1. Auto-Scroll Behavior: Only auto-scroll down if the user is already at the bottom of the list. If they scroll up to read previous messages while tokens are streaming, forcing them back to the bottom will frustrate them.
2. App Lifecycle Transitions: When the app moves to the background (AppLifecycleState.paused), abort active streaming connections immediately. Mobile operating systems will suspend socket threads anyway, and keeping them open wastes background battery and server compute.
3. Local Cache for Offline Reads: Store completed chats in local SQLite or Hive storage. The user should be able to view their previous responses even when boarding a flight or driving through a tunnel.

By separating the mobile UI from the model provider via an edge proxy and streaming with reactive state management, you get fast, responsive AI features without exposing credentials or draining device batteries.