Why Your AI Collaboration Tools Feel Slow (and What Huawei Did About It)

You Know That Lag When Your AI Tool Stalls?

It happens to all of us. You're in a shared coding session, the assistant pauses mid-suggestion, and everyone checks their Wi-Fi. But it's not your network. It's the way the model communicates with itself. In Mixture-of-Experts (MoE) setups, a step called AllToAll can eat up over 30% of the total time. That's the difference between a tool that feels instant and one that makes you want to shut your laptop.

At AICon in Shenzhen (August 21–22), Huawei's Li Baichao will walk through how they've been tackling this. His talk, “Pangu Training and Inference Communication Optimization: Performance Exploration on Ascend,” will share real numbers: a 10% boost in AllToAll performance and a 10% improvement in Time to First Token (TTFT) for ultra-long contexts. For anyone building collab tools, those wins translate directly to a snappier user experience.

Why AllToAll Is Such a Pain

If you've worked with MoE models, you know AllToAll is the grumpy neighbor who always slows down the block party. It shuffles tokens between experts, and it's expensive. In Pangu's expert-parallel (EP) domain, this single operation was the dominant cost. The Huawei team didn't just throw more GPUs at it; they got intimate with the Ascend 950 hardware.

One key insight: the popular DeepEP library, built for NVIDIA GPUs, simply didn't work on Ascend 910A3. That's a reminder that off-the-shelf solutions can't always bridge the hardware gap. Instead, they wrote custom communication operations that exploit Ascend 950's dedicated CCU (Communication Compute Unit) accelerator. The result? A 10% speedup in the EP AllToAll—no small feat when you're shuffling thousands of tokens.

But here's the catch: those optimizations are tailored for Ascend 950. Use them on Ascend 910A2/A3 or NVIDIA H20, and you could see performance nosedive. That trade-off—affinity over generality—is a strategic choice, not a bug.

KV Cache Offloading: A Different Beast

Long-context scenarios—think 1 million tokens—bring a different problem. Here, the bottleneck shifts to Host-to-Device (H2D) transfers of KV cache, which directly impacts TTFT. That's the time it takes for your model to start generating a response, and users hate waiting. Huawei's answer is Omni Cache, a software-hardware co-design that efficiently manages KV cache offloading.

On the hardware side, Ascend 950 gives each NPU a dedicated H2D path. Sounds simple, but it's a game-changer for throughput. On the software side, Omni Cache orchestrates the movement so the GPU isn't starved for data. The combined effect? TTFT improved by over 10%, making those long-context conversations feel almost instant.

One caveat: Omni Cache isn't a silver bullet. It works because they control both the hardware and the software. If you're on a different stack, you'll need to find your own way to hide that latency.

What You Can Learn for Your Own Tools

You might not be running Ascend hardware, but there are still lessons here. First, profile your communication patterns. Where is the latency really coming from? Second, don't assume a library that works on one vendor's chip will work on another. Third, consider custom solutions for your most critical communication hotspots—even a 10% improvement can matter when you're at scale.

Huawei also emphasizes the importance of hiding communication behind computation. The goal is to ensure that communication time is never exposed in the end-to-end latency. That means overlapping data transfers with compute, using multi-stream parallelism, and designing fusion operators that reduce the number of communication steps.

But let's be realistic: not every team has the resources to build custom CCU kernels. For many, the takeaway is simpler—start by measuring your own bottlenecks, then see if there's a quick win like batching smaller transfers or tuning the order of operations.

AICon Shenzhen: More Than Just This Talk

Li Baichao's talk is just one slice of a packed agenda at AICon. With ten dedicated tracks—from AI infrastructure to agent engineering, embodied intelligence, and safety—the conference aims to bridge the gap between model capability and real-world system building. Attendees will hear from 50+ experts from companies like Alibaba, Tencent, Huawei, Kuaishou, and Google Cloud, plus top academics from Zhejiang University.

For anyone involved in building collaborative AI tools, the event is a chance to see how the industry is tackling the engineering challenges that sit between a trained model and a product that actually works. Whether it's making agents more reliable or cutting inference costs, the focus is on practical, hard-won lessons.

Let's Be Honest

Communication optimization might not be the flashiest topic in AI, but it's the unsung hero that determines whether your collaborative tool feels snappy or sluggish. Huawei's work on Ascend 950 shows that with a deep understanding of your hardware, you can make meaningful strides. The path forward isn't about generic solutions; it's about getting your hands dirty with the specifics of your stack.

If you're building the next generation of collaboration platforms, take a moment to look under the hood. Your users might not notice the AllToAll latency, but they'll definitely notice when the tool starts feeling instant.