报名邀请 | 港科大（广州）- 新加坡国立大学人工智能前沿联合论坛

日期时间

2026年7月15日（星期三）

9:00 - 14:30（8:30开始报到）

活动地点

香港科技大学（广州）演讲厅 C

活动报名

主办单位

香港科技大学（广州）人工智能学域

联合主办

新加坡国立大学广州创新研究院

Large language models (LLMs) are rapidly reshaping modern applications, from conversational assistants and code generation to agents with autonomous reasoning capabilities. However, achieving high-throughput, low-cost LLM serving under strict latency SLOs still faces two fundamental challenges: efficient scheduling of GPU-side compute resources, and the latency and cost of cross-region remote data access. This talk will introduce our recent systematic solutions targeting these two layers.

The first part of the talk introduces Libra, a high-performance serving system for imbalanced and dynamic LLM workloads. In real-world scenarios, the variation in prompt and response lengths creates imbalance between the compute-intensive prefill stage and the memory-access-intensive decode stage, causing both traditional PD colocation and PD disaggregation architectures to fall short in simultaneously achieving low tail latency and high throughput. Libra proposes a new Flexible Partitioning and Scheduling (FPS) paradigm, using micro-requests as the core abstraction. It performs fine-grained request partitioning at arbitrary token boundaries and dynamically balances load across a unified GPU pool through a two-level scheduler. Combined with a chunked KV-cache transfer mechanism, Libra improves goodput by up to 1.91× over prior systems on A100/H100 clusters, increases serving capacity by 1.15×–3.07×, and achieves up to 74.2% performance improvement under mixed workloads.

The second part of the talk introduces Cortex, a semantic-aware caching system for cross-region remote data access by LLM agents. LLM agents frequently invoke remote search, RAG, and tool APIs in tasks such as deep research and coding, where cross-region access incurs latencies as high as 300–500 ms and costs on the order of millions of dollars per month. Existing caching approaches rely either on exact matching or similarity comparisons in embedding space, and therefore cannot guarantee semantic correctness. Cortex proposes two core abstractions: Semantic Element (SE) and Semantic Retrieval Index (Seri). By combining ANN-based nearest-neighbor retrieval with a lightweight LLM Semantic Judge for precise semantic verification, Cortex builds a cache pipeline that achieves both high hit rates and correctness guarantees. Together with adaptive eviction, proactive prefetching, and GPU co-location of the main LLM and judge LLM, Cortex achieves a 3.6× throughput improvement and over 85% cache hit rate on search workloads, while maintaining accuracy nearly identical to the no-cache baseline. It also delivers a 20% performance improvement on coding tasks.

This presentation shares two complementary systems to unlock the potential of consumer-grade hardware for LLM serving and tuning. First, SlideFormer enables full-parameter fine-tuning for models up to 123B parameters on a single RTX 4090, with a layer-sliding pipeline and asynchronous offloading to achieve 1.4–6.27× higher throughput. In a multi-GPU data-parallel system, SlideFormer supports finer-grained overlap and elastic checkpointing, achieving over 95% scaling efficiency under near-linear multi-GPU scaling. Second, SMOE enables efficient inference for large-scale MoE models that cannot fit entirely on a single GPU. SMOE combines token-wise expert prefetching with dynamic CPU-GPU co-execution and phase-aware optimization for prefill and decode, yielding 7–8× prefill speedups and up to 20% decoding improvements. SMOE future exploits token-wise prefetching and expert correlation-aware dynamic expert allocation to continuously rebalance expert placement across GPUs, significantly improving resource utilization and achieving notable inference acceleration for large-scale MoE models. Our work collectively demonstrates that with co-designed system optimizations, a single GPU/machine can serve as a powerful, lossless platform for both large-scale LLM training and inference.