在[海光 DCU](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E6%B5%B7%E5%85%89+DCU&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLmtbflhYkgRENVIiwiemhpZGFfc291cmNlIjoiZW50aXR5IiwiY29udGVudF9pZCI6MjgyMTYzNjgzLCJjb250ZW50X3R5cGUiOiJBcnRpY2xlIiwibWF0Y2hfb3JkZXIiOjEsInpkX3Rva2VuIjpudWxsfQ.sSxYFjPNEgezZlKUwmYwL3HNj26lXY5q5PO2Z40GP3Y&zhida_source=entity) 八卡机上，我们的[多模态服务](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%A4%9A%E6%A8%A1%E6%80%81%E6%9C%8D%E5%8A%A1&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLlpJrmqKHmgIHmnI3liqEiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.GZT8LkyfmafpfS_w8IqCkC98bW3hz8WR2p9GwHJB2Hc&zhida_source=entity)最终形态是 8 个 [vLLM](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=vLLM&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJ2TExNIiwiemhpZGFfc291cmNlIjoiZW50aXR5IiwiY29udGVudF9pZCI6MjgyMTYzNjgzLCJjb250ZW50X3R5cGUiOiJBcnRpY2xlIiwibWF0Y2hfb3JkZXIiOjEsInpkX3Rva2VuIjpudWxsfQ.hAJYxI4nEe1ko3MnLle6p_ajr-qy3sfgSpzQmwyCkYY&zhida_source=entity) 实例各占一张卡（端口 8002–8009），对外需要一个统一入口做[负载均衡](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E8%B4%9F%E8%BD%BD%E5%9D%87%E8%A1%A1&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLotJ_ovb3lnYfooaEiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.nzOx7GSP9KmM7htiAs7_L4wZ0IjwYQQuM1opw6K_jfo&zhida_source=entity)。标准答案是 [nginx](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=nginx&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJuZ2lueCIsInpoaWRhX3NvdXJjZSI6ImVudGl0eSIsImNvbnRlbnRfaWQiOjI4MjE2MzY4MywiY29udGVudF90eXBlIjoiQXJ0aWNsZSIsIm1hdGNoX29yZGVyIjoxLCJ6ZF90b2tlbiI6bnVsbH0.WLUTbef5A8fhLXcZJjSWjjUF-IvvEqrm5AAvp3Ge7ec&zhida_source=entity)——但当时的账号没有免密 [sudo](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=sudo&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJzdWRvIiwiemhpZGFfc291cmNlIjoiZW50aXR5IiwiY29udGVudF9pZCI6MjgyMTYzNjgzLCJjb250ZW50X3R5cGUiOiJBcnRpY2xlIiwibWF0Y2hfb3JkZXIiOjEsInpkX3Rva2VuIjpudWxsfQ.MyIjKlEd8pkvZTu2pDBpRwfijrWR-tp9-8UhfUzaZWY&zhida_source=entity)，装不了、也改不了配置。 于是用 aiohttp 手写了一个[反向代理](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%8F%8D%E5%90%91%E4%BB%A3%E7%90%86&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLlj43lkJHku6PnkIYiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.gAb0zCwX1l-tCT3oJkVZfCsD7cvM8ueNKGHQD29I8zE&zhida_source=entity)。核心逻辑不到 100 行，在生产流量下跑了几周，期间按真实故障迭代出了一堆「教科书上不写」的设计决策。这篇把完整实现和每个决策的为什么都摊开——读者可以直接抄走用。 先明确这个组件的需求清单： | 能力 | 为什么需要 | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | 最少连接调度 | 8 张卡对等地分摊请求，不能让某张卡排队别卡空闲 | | 周期健康检查 | 实例崩了要自动摘除，恢复了要自动加回 | | [SSE 流式透传](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=SSE+%E6%B5%81%E5%BC%8F%E9%80%8F%E4%BC%A0&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJTU0Ug5rWB5byP6YCP5LygIiwiemhpZGFfc291cmNlIjoiZW50aXR5IiwiY29udGVudF9pZCI6MjgyMTYzNjgzLCJjb250ZW50X3R5cGUiOiJBcnRpY2xlIiwibWF0Y2hfb3JkZXIiOjEsInpkX3Rva2VuIjpudWxsfQ.eSGHMofoRxu3Bpn00Hk6kNdDryZgr6W0m_dLdrBxuLc&zhida_source=entity) | vLLM 的流式输出必须逐[chunk](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=chunk&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJjaHVuayIsInpoaWRhX3NvdXJjZSI6ImVudGl0eSIsImNvbnRlbnRfaWQiOjI4MjE2MzY4MywiY29udGVudF90eXBlIjoiQXJ0aWNsZSIsIm1hdGNoX29yZGVyIjoxLCJ6ZF90b2tlbiI6bnVsbH0.h5SYsRpd9OI9Sd0neAmhC7P4KAWQfO9emoeg0VCNG7A&zhida_source=entity)转发，不能攒完再回 | | 请求追踪 | 每个响应标明命中了哪个后端，排查有据可查 | | 结果日志 | 记录每个请求的[元数据](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%85%83%E6%95%B0%E6%8D%AE&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLlhYPmlbDmja4iLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.TGvso3A7nRnH38uLuDf2IKhMkjU5LV2ckpidX_2n22w&zhida_source=entity)（详见下文，排查[生命线](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E7%94%9F%E5%91%BD%E7%BA%BF&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLnlJ_lkb3nur8iLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.vrahp0UQbeu46Gp1uTUXMtWg-iS_YqQC6DPWoMiohhY&zhida_source=entity)） | 以及一个最重要的**[非功能需求](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E9%9D%9E%E5%8A%9F%E8%83%BD%E9%9C%80%E6%B1%82&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLpnZ7lip_og73pnIDmsYIiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.cBb8ho1tP0neLzCmHCcmmGH2CqkylRkFpnF3x9j1Bc4&zhida_source=entity)：可退役**——它是无 sudo 环境下的替代品，将来拿到 root 换 nginx 时，客户端必须零改动。这意味着所有客户端可见的行为（入口地址、路径、协议）都要按「标准反向代理」的语义实现，不夹带私货。 ## 骨架：调度与探活 ```text import asyncio, json, os from collections import defaultdict from aiohttp import web, ClientSession, ClientTimeout LB_PORT = int(os.environ.get("LB_PORT", "8000")) HC_INTERVAL = int(os.environ.get("LB_HC_INTERVAL", "10")) _default = [f"127.0.0.1:{p}" for p in range(8002, 8010)] BACKEND_URLS = [f"http://{b}" for b in os.environ.get("LB_BACKENDS", "").split(",") if b] or _default healthy = {u: False for u in BACKEND_URLS} active = defaultdict(int) # 各后端 in-flight 请求数 _rr = 0 # 平局时的轮询游标 def pick_backend(): """最少连接；平局按轮询。无健康后端返回 None。""" global _rr cand = [u for u in BACKEND_URLS if healthy[u]] if not cand: return None min_n = min(active[u] for u in cand) tied = [u for u in cand if active[u] == min_n] u = tied[_rr % len(tied)] _rr += 1 return u ``` 探活是一个常驻协程： ```text async def health_check(session: ClientSession): async def probe(u): try: async with session.get(f"{u}/health", timeout=ClientTimeout(total=5)) as r: healthy[u] = (r.status == 200) except Exception: healthy[u] = False while True: await asyncio.gather(*(probe(u) for u in BACKEND_URLS)) await asyncio.sleep(HC_INTERVAL) ``` ## 决策一：探活用 `/health`，不用 `/v1/models` 这是[迭代](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=2&q=%E8%BF%AD%E4%BB%A3&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLov63ku6MiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MiwiemRfdG9rZW4iOm51bGx9.no8RaPfBzu8eQeuCrTLsoVX4y7IsMWs-6lYzcQPrBvs&zhida_source=entity)出来的第一个教训。最初探的是 `/v1/models`，看起来一切正常——直到某次发现请求还在往一个已经崩掉的实例上打。原因：vLLM 的 `/v1/models` 只反映 **API Server 层**，而真正干活的是 EngineCore 子进程；EngineCore 崩了，API Server 还活着，`/v1/models` 照样返回 200。负载均衡于是忠实地把流量分给了一具尸体。 `/health` 直接反映引擎状态：模型加载中返回 503，EngineCore 死了也返回 503。**探活端点的选择标准不是「它能不能返回 200」，而是「它 503 的时候是不是真的死了」**。 ## 决策二：in-flight 计数的时机，差一行就是雪崩 最少连接调度依赖 `active[backend]` 计数。第一版把计数放在「读完请求 body 之后」： ```text backend = pick_backend() body = await request.read() # ← 高并发下这里有毫秒级时间窗 active[backend] += 1 # ← 计数太晚 ``` 高并发下，几十个[协程](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=2&q=%E5%8D%8F%E7%A8%8B&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLljY_nqIsiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MiwiemRfdG9rZW4iOm51bGx9.b60UqnRSGglD-v0phuqliAYEqfCPBCbGC9wwV4P6H9w&zhida_source=entity)同时在 `read()` 处等待，全都看到同一个后端「当前最闲」（计数还是 0），然后一起涌过去。实测单卡最大堆积 22 个请求（该实例 `max_seqs` 只有 1，其余全在排队），而其它七张卡在睡觉——调度器亲手制造了[负载不均](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E8%B4%9F%E8%BD%BD%E4%B8%8D%E5%9D%87&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLotJ_ovb3kuI3lnYciLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.72gjxGlsuKnsQ-XBaHPuMv65yno4t8gR5QPvwg4c1Vo&zhida_source=entity)。 修复是把计数提到 `pick_backend()` 之后、`read()` 之前，并用 `try/finally` 保证任何[异常路径](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%BC%82%E5%B8%B8%E8%B7%AF%E5%BE%84&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLlvILluLjot6_lvoQiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.QlRxjcmhEWcIGpibxnQOVBeUZHk6hiM7LZDVVf590ck&zhida_source=entity)都会减计数： ```text backend = pick_backend() active[backend] += 1 # ← 先占坑 upstream = None try: body = await request.read() ...转发... finally: active[backend] -= 1 # ← 任何路径都回退 if upstream is not None: upstream.close() ``` 一行代码换个位置，最大堆积从 22 降到 5。**共享可变状态的更新时机，和更新本身一样重要。** ## 决策三：什么情况才摘后端 转发失败有十几种姿势，但「摘除后端」是个重操作（要等下一个健康检查周期才回来），不能滥用。最终策略是**只对「连不上」摘除**： ```text except (ClientConnectorError, ConnectionError, OSError) as e: healthy[backend] = False # 进程挂/端口没监听 → 摘 return web.json_response({"error": f"backend unreachable: {e}"}, status=502) except asyncio.TimeoutError: return web.json_response({"error": "backend timeout"}, status=504) # 不摘 except Exception as e: return web.json_response({"error": f"proxy error: {e}"}, status=502) # 不摘 ``` 理由：超时可能是单请求太大（`max_tokens` 给狠了），读中断可能是客户端主动取消——这些和后端健康无关，摘了反而误伤。只有 TCP 层连不上，才是后端真死的证据。 ## SSE 流式透传 vLLM 的[流式响应](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E6%B5%81%E5%BC%8F%E5%93%8D%E5%BA%94&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLmtYHlvI_lk43lupQiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.rGZg0OHBq9sA60K8OcE9YbUShp7mEUCPmMt6NL9rI4M&zhida_source=entity)是 `text/event-stream`，必须逐 chunk 转发（攒完再回会失去流式意义，客户端超时）。核心就是 `iter_any()` + 边收边写： ```text resp = web.StreamResponse(status=upstream.status, headers=过滤后的响应头) if upstream.headers.get("Content-Type", "").startswith("text/event-stream"): resp.content_type = "text/event-stream" resp.headers["X-Backend"] = backend.replace("http://", "") await resp.prepare(request) async for chunk in upstream.content.iter_any(): await resp.write(chunk) await resp.write_eof() ``` 顺手把 `X-Backend` / `X-Backend-Port` 两个响应头加上——每个响应标明命中了哪张卡。排查「某几张图的结果不对」时，凭这个头可以直接定位到具体实例的日志。 ## 结果日志：每个请求一行 转发的同时，把响应的前 512KB 缓进 `buf`（大响应截断），对 `/chat/completions` 的请求解析出[结构化](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E7%BB%93%E6%9E%84%E5%8C%96&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLnu5PmnoTljJYiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.uffWIsGSLdW5Uin9SspCXNVA4UW1svZD-CYKVdm0wFQ&zhida_source=entity)摘要记一行日志： ```text 08-11 14:22:01 orid=ORD-123 127.0.0.1:8003 status=200 finish=stop ctok=432/1024 thinking=False think=True empty=False clen=2107 | 图片显示一位年长的男性… 08-11 14:22:03 orid=905112 127.0.0.1:8005 status=400 req_keys=['max_tokens','messages', 'model','stream','temperature','user'] ERR_BODY | {"error":...} ``` 字段设计每个都有用途：`orid`（业务追踪 ID，优先取请求头 `X-Orid`，兜底 body 的 `user` 字段）用来和业务方对账；`finish`+`ctok/max_tokens` 判断截断；`think`/`empty` 判断输出异常；**非 200 的请求把后端返回的错误正文完整记下来**——这一点极其关键，因为 vLLM 自己的 access 日志只记状态码不记 body，400 的具体原因它不写，你不抓就永远不知道。 [解析函数](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E8%A7%A3%E6%9E%90%E5%87%BD%E6%95%B0&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLop6PmnpDlh73mlbAiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.438edUCyzxOGEZ7uGbp91ks5MYvqdB21AhcN9MB4cTk&zhida_source=entity)要同时兼容非流式 JSON 和 SSE 流（SSE 就把所有 `data:` 行拼起来取 `delta.content` 和最后的 `finish_reason`/`usage`）。日志用 `RotatingFileHandler` 做轮转（默认 50MB×3 份）——这个是补课：第一版没做轮转，几周后这条日志悄悄涨到了 769MB。 ## 运行与验证 ```text python3 lb_proxy.py & # 监听 :8000 curl -s http://127.0.0.1:8000/__lb_status # {"healthy": 8, "total": 8, "backends": [{"backend": "127.0.0.1:8002", # "healthy": true, "active": 0}, ...]} ``` 自带一个 `/__lb_status` 状态端点，输出每个后端的健康与 in-flight 数——集群巡检一眼扫完。压测下这个「土法」代理扛到了每秒 60+ 请求转发，自身从不是瓶颈（瓶颈永远在 GPU）。 ## 边界与局限 说清楚它不管什么：[单进程](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%8D%95%E8%BF%9B%E7%A8%8B&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLljZXov5vnqIsiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.R2ceeAc4_pjylwDAA1V0w0f-fZPcLJTmR_mqBNgfk_k&zhida_source=entity)单点（代理挂了全挂，生产上应配 supervisor 或 systemd——我们用 [nohup](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=nohup&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiJub2h1cCIsInpoaWRhX3NvdXJjZSI6ImVudGl0eSIsImNvbnRlbnRfaWQiOjI4MjE2MzY4MywiY29udGVudF90eXBlIjoiQXJ0aWNsZSIsIm1hdGNoX29yZGVyIjoxLCJ6ZF90b2tlbiI6bnVsbH0.avgY8aSn8MlE6ZQNuQqBP3sPVaGxqTR3lWROZ4WD7b0&zhida_source=entity) + 监控脚本顶着）；无 TLS、无鉴权（内网场景够用）；无[动态路由](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E5%8A%A8%E6%80%81%E8%B7%AF%E7%94%B1&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLliqjmgIHot6_nlLEiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.MA3s8S-5g67-T30Sh1YLj-l9A01rFThYiwJXkdzxJPw&zhida_source=entity)、无灰度（用不上）。**当你的需求超出这张清单，就是换 nginx/haproxy 的时候**——而这个组件的[设计目标](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E8%AE%BE%E8%AE%A1%E7%9B%AE%E6%A0%87&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLorr7orqHnm67moIciLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.piPdp52Co3-WFgK4dKMd69iVfUUfiuMXqozYpqCDocE&zhida_source=entity)本来就是：那一天到来时，改一条 DNS 或[端口映射](https://zhida.zhihu.com/search?content_id=282163683&content_type=Article&match_order=1&q=%E7%AB%AF%E5%8F%A3%E6%98%A0%E5%B0%84&zd_token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ6aGlkYV9zZXJ2ZXIiLCJleHAiOjE3OTA4NDE3NjksInEiOiLnq6_lj6PmmKDlsIQiLCJ6aGlkYV9zb3VyY2UiOiJlbnRpdHkiLCJjb250ZW50X2lkIjoyODIxNjM2ODMsImNvbnRlbnRfdHlwZSI6IkFydGljbGUiLCJtYXRjaF9vcmRlciI6MSwiemRfdG9rZW4iOm51bGx9.VzdFGkaVQgO6k8OOPrace-x8C29JWCKjphpS82TXGNY&zhida_source=entity)就能退役它，客户端一行不用动。 ## 小结 100 行代码做负载均衡，难的不是写出来，而是三个「只有生产才会教你」的决策：探活端点要选「死了真会说」的那个；共享计数要在时间窗关闭后更新；摘后端要只信物理层的死亡证据。这三个决策加上结果日志，构成了一个没有 root 权限下完全可用的生产组件——也构成了下次有人问你「为什么不能随便拿个 HTTP 转发脚本当负载均衡」时的答案。 --- *环境信息：海光 K100\_AI × 8 / 8 × vLLM 实例（8002–8009）/ aiohttp 3.14 反向代理 :8000 / 生产流量运行数周，峰值转发 60+ r/s。*