Getting started with LLMs and its on-device deployment

Intro

Large Language Models (LLMs) for edge computing refers to deploying these powerful AI models directly on devices at the edge of a network rather than relying on centralized cloud servers.

What are LLMs?

Large language models (LLMs) are artificial intelligence, mainly natural language processing (NLP) models, trained on massive amounts of text data. This allows them to perform various tasks, such as Generating Text, Translating Languages, Answering Questions, etc.

Getting Started with LLMs

Below are some of the resources you can use to get started with LLMs based on your level of familiarity with Machine learning and Natural language processing:

• Developing LLMs-powered applications: Build AI Apps with ChatGPT, Dall-E, and GPT-4
• ML Engineer tasks such as fine-turning: Generative AI with Large Language Models
• Research Level: Natural Language Processing with Attention Models

Benefits and Challenges of Deploying AI Models on-device

Benefits of LLMs to Edge

• Reduced latency: Edge computing minimizes the distance data needs to travel, leading to faster response times and real-time insights. This is crucial for applications like autonomous vehicles, robotics, and virtual reality.
• Improved privacy and security: By keeping data processing on edge, sensitive information doesn't need to be sent to the cloud and does not involve associated risks.
• Offline functionality: If LLMs are on-edge devices, they can still function without an internet connection, making them ideal for critical infrastructure.

Challenges of LLMs to Edge

• Hardware Limitations: Edge devices typically have less processing power and memory than cloud servers, requiring efficient LLM models and techniques.
• Energy Consumption: Running LLMs on edge devices, especially resource-intensive ones, can increase power consumption.
• Limited Update and Maintenance: Keeping LLMs on edge devices up to date requires efficient update mechanisms and robust infrastructure.

Bridging Edge AI and LLM Integration

Model Quantization

LLMs are generally quite large and computationally expensive. They need ample memory and processing power, often exceeding the capabilities of edge devices. This makes it difficult to directly deploy these powerful models on the edge for real-time applications.

Deploying LLMs on edge devices requires selecting smaller, optimized models tailored to specific use cases, ensuring smooth operation within limited resources.

Quantization is a technique to reduce the model size. It aims to reduce the size and computational demands of the LLM model without significantly affecting its accuracy performance.

Quantization reduces model precision, converting parameters from 32-bit floats to lower-precision formats like 16-bit floats or 8-bit integers. This involves mapping high-precision values to a smaller range with scale and offset adjustments, which saves memory and speeds up computations. It reduces hardware costs and energy consumption while maintaining real-time performance like NLP.

This makes LLMs feasible for resource-constrained devices like mobile phones and edge platforms. AI tools like TensorFlow, PyTorch, and Qualcomm's AIMET library provide advanced quantization and compression techniques to optimize models for different frameworks and needs.

Benefits of Model Quantization for LLMs on Edge:

• Reduced Model Size: Smaller models require less storage space and bandwidth, making them easier to download and deploy on edge devices.
• Faster Inference: Lower-precision computations lead to quicker predictions and responses, crucial for real-time applications at the edge.
• Lower Power Consumption: Reduced processing requirements translate to less power consumption.

Quantization inevitably introduces some information loss, which can impact the accuracy of the model's predictions. Depending on the specific application, this trade-off between efficiency and accuracy needs careful consideration.

Various quantization techniques

Post-Training Quantization (PTQ): Reduces the precision of weights in a pre-trained model after training, converting them to 8-bit integers or 16-bit floating-point numbers.

Quantization-Aware Training (QAT): Integrates quantization during training, allowing weight adjustments for lower precision.

Zero-Shot Post-Training Uniform Quantization: Applies standard quantization without further training, assessing its impact on various models.

Weight-Only Quantization: Focuses only on weights, converting them to FP16 during matrix multiplication to improve inference speed and reduce data loading.

Pruning and Quantization

Pruning reduces redundant neurons and connections in an AI model. It analyses the network using weight magnitude (assumes that smaller weights contribute less to the output) or sensitivity analysis methods (how much the model’s output changes when a specific weight is altered) to determine which parts have minimal impact on the final predictions. They are then either removed, or their weights are set to zero. After pruning, the model may be fine-tuned to recover any performance lost during the pruning process. Major techniques for pruning are Structured pruning and Unstructured pruning.

Low-Rank Adaptation (LoRA)

LoRA compresses models by decomposing weight matrices into lower-dimensional components, reducing the number of trainable parameters while maintaining accuracy. It allows for efficient fine-tuning and task-specific adaptation without full retraining.

AI tools integrate LLMs with LoRA by adding low-rank matrices to the model architecture, reducing trainable parameters, and enabling efficient fine-tuning. Tools like Loralib simplify it, making model customization cost-effective and resource-efficient. For instance, LoRA minimizes the number of trainable parameters in large models like LLaMA-70B, significantly lowering GPU memory usage. It allows LLMs to operate efficiently on edge devices with limited resources, enabling real-time processing and reducing dependence on cloud infrastructure.

On-device Inference

In an example of on-device inference, lightweight models like Gemma-2B, Phi-2, and StableLM-3B were successfully run on an Android device using TensorFlow Lite and MediaPipe. Quantizing these models reduced their size and computational demands, making them suitable for edge devices. After transferring the quantized model to an Android phone and adjusting the app’s code, testing on a Snapdragon 778 chipset showed that the Gemma-2B model could generate responses in seconds. This demonstrates how quantization and on-device inference enable efficient LLM performance on mobile devices.

Hybrid Inference

Hybrid inference combines edge and cloud resources, distributing model computations to balance performance and resource constraints. This approach allows resource-intensive tasks to be handled by the cloud while latency-sensitive tasks are managed locally on the edge device.