如何处理速率限制
当您反复调用OpenAI API时，可能会遇到错误消息，提示“429：‘请求过多’”或“RateLimitError”。这些错误消息来自超出API的速率限制。
本指南分享了避免和处理速率限制错误的技巧。
有关用于限制并行请求以避免速率限制错误的示例脚本，请参阅api_request_parallel_processor.py。
为什么存在速率限制
速率限制是API的常见做法，它们的存在有几个不同的原因。
- 首先，它们有助于防止API被滥用或误用。例如，恶意行为者可能会向API发送大量请求，试图使其过载或导致服务中断。通过设置速率限制，OpenAI可以阻止此类活动。
- 其次，速率限制有助于确保每个人都能公平地访问API。如果一个人或组织发出了过多的请求，可能会拖累所有其他人的API。通过限制单个用户可以发出的请求数量，OpenAI确保每个人都有机会使用API而不会遇到速度变慢的情况。
- 最后，速率限制可以帮助OpenAI管理其基础设施的总负载。如果API请求急剧增加，可能会给服务器带来压力并导致性能问题。通过设置速率限制，OpenAI可以帮助所有用户保持平稳一致的体验。
尽管遇到速率限制可能会令人沮丧，但速率限制的存在是为了保护API对其用户的可靠运行。
默认速率限制
您的速率限制和支出限制（配额）会根据多种因素自动调整。随着您对OpenAI API的使用量增加并成功支付账单，我们会自动提高您的使用层级。您可以使用以下资源找到有关速率限制的具体信息。
其他速率限制资源
在其他资源中阅读有关OpenAI速率限制的更多信息：
请求增加速率限制
要了解有关提高组织使用层级和速率限制的更多信息，请访问您的限制设置页面。
import openai
import os
client = openai.OpenAI(api_key=os.getenv("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
示例速率限制错误
当API请求发送过快时，将发生速率限制错误。如果使用OpenAI Python库，它们看起来会像这样：
RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.
下面是触发速率限制错误的示例代码。
# 在循环中请求大量补全
for _ in range(100):
client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Hello"}],
max_tokens=10,
)
---------------------------------------------------------------------------
RateLimitError Traceback (most recent call last)
Cell In[2], line 3
1 # 在循环中请求大量补全
2 for _ in range(100):
----> 3 client.chat.completions.create(**kwargs)
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_utils/_utils.py:279, in required_args.<locals>.inner.<locals>.wrapper(*args, **kwargs)
277 msg = f"Missing required argument: {quote(missing[0])}"
278 raise TypeError(msg)
--> 279 return func(*args, **kwargs)
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/resources/chat/completions.py:859, in Completions.create(self, messages, model, audio, frequency_penalty, function_call, functions, logit_bias, logprobs, max_completion_tokens, max_tokens, metadata, modalities, n, parallel_tool_calls, prediction, presence_penalty, reasoning_effort, response_format, seed, service_tier, stop, store, stream, stream_options, temperature, tool_choice, tools, top_logprobs, top_p, user, extra_headers, extra_query, extra_body, timeout)
817 @required_args(["messages", "model"], ["messages", "model", "stream"])
818 def create(
819 self,
(...)
856 timeout: float | httpx.Timeout | None | NotGiven = NOT_GIVEN,
857 ) -> ChatCompletion | Stream[ChatCompletionChunk]:
858 validate_response_format(response_format)
--> 859 return self._post(
860 "/chat/completions",
861 body=maybe_transform(
862 {
863 "messages": messages,
864 "model": model,
865 "audio": audio,
866 "frequency_penalty": frequency_penalty,
867 "function_call": function_call,
868 "functions": functions,
869 "logit_bias": logit_bias,
870 "logprobs": logprobs,
871 "max_completion_tokens": max_completion_tokens,
872 "max_tokens": max_tokens,
873 "metadata": metadata,
874 "modalities": modalities,
875 "n": n,
876 "parallel_tool_calls": parallel_tool_calls,
877 "prediction": prediction,
878 "presence_penalty": presence_penalty,
879 "reasoning_effort": reasoning_effort,
880 "response_format": response_format,
881 "seed": seed,
882 "service_tier": service_tier,
883 "stop": stop,
884 "store": store,
885 "stream": stream,
886 "stream_options": stream_options,
887 "temperature": temperature,
888 "tool_choice": tool_choice,
889 "tools": tools,
890 "top_logprobs": top_logprobs,
891 "top_p": top_p,
892 "user": user,
893 },
894 completion_create_params.CompletionCreateParams,
895 ),
896 options=make_request_options(
897 extra_headers=extra_headers, extra_query=extra_query, extra_body=extra_body, timeout=timeout
898 ),
899 cast_to=ChatCompletion,
900 stream=stream or False,
901 stream_cls=Stream[ChatCompletionChunk],
902 )
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1283, in SyncAPIClient.post(self, path, cast_to, body, options, files, stream, stream_cls)
1269 def post(
1270 self,
1271 path: str,
(...)
1278 stream_cls: type[_StreamT] | None = None,
1279 ) -> ResponseT | _StreamT:
1280 opts = FinalRequestOptions.construct(
1281 method="post", url=path, json_data=body, files=to_httpx_files(files), **options
1282 )
-> 1283 return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:960, in SyncAPIClient.request(self, cast_to, options, remaining_retries, stream, stream_cls)
957 else:
958 retries_taken = 0
--> 960 return self._request(
961 cast_to=cast_to,
962 options=options,
963 stream=stream,
964 stream_cls=stream_cls,
965 retries_taken=retries_taken,
966 )
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1049, in SyncAPIClient._request(self, cast_to, options, retries_taken, stream, stream_cls)
1047 if remaining_retries > 0 and self._should_retry(err.response):
1048 err.response.close()
-> 1049 return self._retry_request(
1050 input_options,
1051 cast_to,
1052 retries_taken=retries_taken,
1053 response_headers=err.response.headers,
1054 stream=stream,
1055 stream_cls=stream_cls,
1056 )
1058 # If the response is streamed then we need to explicitly read the response
1059 # to completion before attempting to access the response text.
1060 if not err.response.is_closed:
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1098, in SyncAPIClient._retry_request(self, options, cast_to, retries_taken, response_headers, stream, stream_cls)
1094 # In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a
1095 # different thread if necessary.
1096 time.sleep(timeout)
-> 1098 return self._request(
1099 options=options,
1100 cast_to=cast_to,
1101 retries_taken=retries_taken + 1,
1102 stream=stream,
1103 stream_cls=stream_cls,
1104 )
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1049, in SyncAPIClient._request(self, cast_to, options, retries_taken, stream, stream_cls)
1047 if remaining_retries > 0 and self._should_retry(err.response):
1048 err.response.close()
-> 1049 return self._retry_request(
1050 input_options,
1051 cast_to,
1052 retries_taken=retries_taken,
1053 response_headers=err.response.headers,
1054 stream=stream,
1055 stream_cls=stream_cls,
1056 )
1058 # If the response is streamed then we need to explicitly read the response
1059 # to completion before attempting to access the response text.
1060 if not err.response.is_closed:
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1098, in SyncAPIClient._retry_request(self, options, cast_to, retries_taken, response_headers, stream, stream_cls)
1094 # In a synchronous context we are blocking the entire thread. Up to the library user to run the client in a
1095 # different thread if necessary.
1096 time.sleep(timeout)
-> 1098 return self._request(
1099 options=options,
1100 cast_to=cast_to,
1101 retries_taken=retries_taken + 1,
1102 stream=stream,
1103 stream_cls=stream_cls,
1104 )
File ~/code/openai-cookbook/.venv/lib/python3.9/site-packages/openai/_base_client.py:1064, in SyncAPIClient._request(self, cast_to, options, retries_taken, stream, stream_cls)
1061 err.response.read()
1063 log.debug("Re-raising status error")
-> 1064 raise self._make_status_error_from_response(err.response) from None
1066 return self._process_response(
1067 cast_to=cast_to,
1068 options=options,
(...)
1072 retries_taken=retries_taken,
1073 )
RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}
如何缓解速率限制错误
使用指数退避进行重试
缓解速率限制错误的一种简单方法是自动重试具有随机指数退避的请求。使用指数退避进行重试意味着在遇到速率限制错误时执行短暂的睡眠，然后重试不成功的请求。如果请求仍然不成功，则增加睡眠时间并重复该过程。此过程将一直持续到请求成功或达到最大重试次数为止。
这种方法有很多好处：
- 自动重试意味着您可以从速率限制错误中恢复，而不会崩溃或丢失数据
- 指数退避意味着您的第一次重试可以快速尝试，同时在您的前几次重试失败时仍能从更长的延迟中受益
- 添加随机抖动以延迟有助于避免重试同时发生
请注意，不成功的请求会计入您的每分钟限制，因此连续发送请求将不起作用。
以下是一些示例解决方案。
示例 #1：使用 Tenacity 库
Tenacity 是一个 Apache 2.0 许可的通用重试库，用 Python 编写，可以简化为几乎任何东西添加重试行为的任务。
要为您的请求添加指数退避，您可以使用 tenacity.retry 装饰器。以下示例使用 tenacity.wait_random_exponential 函数为请求添加随机指数退避。
请注意，Tenacity 库是第三方工具，OpenAI 对其可靠性或安全性不作任何保证。
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
) # 用于指数退避
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_with_backoff(**kwargs):
return client.chat.completions.create(**kwargs)
completion_with_backoff(model="gpt-4o-mini", messages=[{"role": "user", "content": "Once upon a time,"}])
ChatCompletion(id='chatcmpl-8PAu6anX2JxQdYmJRzps38R8u0ZBC', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content='in a small village nestled among green fields and rolling hills, there lived a kind-hearted and curious young girl named Lily. Lily was known for her bright smile and infectious laughter, bringing joy to everyone around her.\n\nOne sunny morning, as Lily played in the meadows, she stumbled upon a mysterious book tucked away beneath a tall oak tree. Intrigued, she picked it up and dusted off its weathered cover to reveal intricate golden patterns. Without hesitation, she opened it, discovering that its pages were filled with magical tales and enchanting adventures.\n\nAmong the stories she found, one particularly caught her attention—a tale of a long-lost treasure hidden deep within a mysterious forest. Legend had it that whoever found this hidden treasure would be granted one wish, no matter how big or small. Excited by the prospect of finding such treasure and fulfilling her wildest dreams, Lily decided to embark on a thrilling journey to the forest.\n\nGathering her courage, Lily told her parents about the magical book and her quest to find the hidden treasure. Though concerned for their daughter\'s safety, they couldn\'t help but admire her spirit and determination. They hugged her tightly and blessed her with love and luck, promising to await her return.\n\nEquipped with a map she found within the book, Lily ventured into the depths of the thick forest. The trees whispered tales of forgotten secrets, and the enchanted creatures hidden within watched her every step. But Lily remained undeterred, driven by her desire to discover what lay ahead.\n\nDays turned into weeks as Lily traversed through dense foliage, crossed swift rivers, and climbed treacherous mountains. She encountered mystical beings who offered guidance and protection along her perilous journey. With their help, she overcame countless obstacles and grew braver with each passing day.\n\nFinally, after what felt like an eternity, Lily reached the heart of the forest. There, beneath a jeweled waterfall, she found the long-lost treasure—a magnificent chest adorned with sparkling gemstones. Overwhelmed with excitement, she gently opened the chest to reveal a brilliant light that illuminated the forest.\n\nWithin the glow, a wise voice echoed, "You have proven your courage and pure heart, young Lily. Make your wish, and it shall be granted."\n\nLily thought deeply about her wish, realizing that her true treasure was the love and happiness she felt in her heart. Instead of making a wish for herself, she asked for the wellbeing and prosperity of her village, spreading joy and harmony to everyone living there.\n\nAs the light faded, Lily knew her quest was complete. She retraced her steps through the forest, returning home to find her village flourishing. Fields bloomed with vibrant flowers, and laughter filled the air.\n\nThe villagers greeted Lily with open arms, recognizing her selflessness and the magic she had brought into their lives. From that day forward, they told the tale of Lily\'s journey, celebrating her as a heroine who embodied the power of love, kindness, and the belief that true treasure lies within oneself.\n\nAnd so, the story of Lily became an everlasting legend, inspiring generations to follow their dreams, be selfless, and find the true treasures that lie within their hearts.', role='assistant', function_call=None, tool_calls=None))], created=1701010806, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=641, prompt_tokens=12, total_tokens=653))
示例 #2：使用 backoff 库
另一个提供用于退避和重试的函数装饰器的库是backoff。
与 Tenacity 一样，backoff 库是第三方工具，OpenAI 对其可靠性或安全性不作任何保证。
import backoff # 用于指数退避
@backoff.on_exception(backoff.expo, openai.RateLimitError, max_time=60, max_tries=6)
def completions_with_backoff(**kwargs):
return client.chat.completions.create(**kwargs)
completions_with_backoff(model="gpt-4o-mini", messages=[{"role": "user", "content": "Once upon a time,"}])
ChatCompletion(id='chatcmpl-AqRiD3gF3q8VVs6w8jgba6FHGr0L5', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content="in a small village nestled between lush green hills and a shimmering lake, there lived a young girl named Elara. Elara had a curious spirit and a heart full of dreams. She often spent her days wandering the woods, exploring the enchanted forest, and collecting wildflowers.\n\nOne sunny afternoon, while wandering deeper into the forest than she ever had before, Elara stumbled upon a sparkling, crystal-clear pond. As she knelt down to take a closer look, she noticed a glimmering object at the bottom. It was a beautifully crafted key, shining with an otherworldly light. Without thinking twice, Elara reached into the cool water and retrieved the key, feeling a strange warmth envelop her.\n\nLittle did she know, this key was no ordinary key. It was said to unlock a secret door hidden in the heart of the forest, a door that led to a realm of wonder and adventure. Legends whispered of enchanted beings, ancient wisdom, and challenges that could only be overcome through bravery and kindness.\n\nExcited by the possibility of what awaited her, Elara set off on a quest to find the hidden door. Guided by a faint glow that seemed to beckon her, she journeyed through twisting pathways, lush groves, and enchanted glades.\n\nAlong the way, she encountered talking animals, wise old trees, and mischievous fairies, each offering clues and riddles that tested her resolve and imagination. With each challenge she faced, Elara grew stronger and more confident, realizing that the true magic lay not just in the world around her, but within herself.\n\nAfter what felt like days of exploring, she finally found the door—a majestic archway covered in vines and blossoms, with a keyhole that sparkled like the night sky. Heart pounding with excitement, Elara inserted the key. With a gentle turn, the door slowly creaked open, revealing a land more breathtaking than she could have ever imagined.\n\nAs she stepped through the doorway, she found herself in a vibrant world filled with colors beyond description, where the sky shimmered in hues of gold and lavender, and the air was filled with the sweet scent of flowers that sang as they swayed in the breeze. Here, she encountered beings of light who welcomed her with open arms.\n\nBut soon, she discovered that this realm was in peril. A dark shadow loomed over the land, threatening to steal its magic and joy. Elara knew she couldn’t stand by and do nothing. With the friends she had made along her journey and the courage she had found within herself, she set out to confront the darkness.\n\nThrough trials that tested her strength, intellect, and compassion, Elara and her friends gathered the forgotten magic of the realm. They united their powers, confronting the shadow in an epic battle of light and dark. In the end, it was Elara's unwavering belief in hope and friendship that banished the darkness, restoring peace and harmony to the land.\n\nGrateful for her bravery, the beings of light gifted Elara a shimmering pendant that would allow her to return to their world whenever she wished, reminding her that true magic lies in the connections we forge with others and the courage to follow our dreams.\n\nWith her heart full of joy, Elara returned to her village, forever changed by her adventure. She would often revisit the magical realm, sharing stories with her friends and inspiring them to embrace their own dreams. And so, the girl who once wandered the woods became a beacon of hope, a reminder that within every heart lies the power to change the world.\n\nAnd from that day on, the little village thrived, full of laughter, love, and dreams waiting to be explored—each adventure beginning just like hers, with a curious heart and a willingness to believe in the impossible. \n\nThe end.", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 794, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 795, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 767, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.20319080352783203, 'batcher_initial_queue_time': 0.12981152534484863}])], created=1737062945, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_72ed7ab54c', usage=CompletionUsage(completion_tokens=767, prompt_tokens=12, total_tokens=779, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))
示例 3：手动实现退避
如果您不想使用第三方库，可以实现自己的退避逻辑。
# 导入
import random
import time
# 定义重试装饰器
def retry_with_exponential_backoff(
func,
initial_delay: float = 1,
exponential_base: float = 2,
jitter: bool = True,
max_retries: int = 10,
errors: tuple = (openai.RateLimitError,),
):
"""使用指数退避重试函数。"""
def wrapper(*args, **kwargs):
# 初始化变量
num_retries = 0
delay = initial_delay
# 循环直到成功响应或达到最大重试次数或引发异常
while True:
try:
return func(*args, **kwargs)
# 重试指定的错误
except errors as e:
# 增加重试次数
num_retries += 1
# 检查是否已达到最大重试次数
if num_retries > max_retries:
raise Exception(
f"已超过最大重试次数 ({max_retries})。"
)
# 增加延迟
delay *= exponential_base * (1 + jitter * random.random())
# 睡眠延迟时间
time.sleep(delay)
# 针对任何未指定的错误引发异常
except Exception as e:
raise e
return wrapper
@retry_with_exponential_backoff
def completions_with_backoff(**kwargs):
return client.chat.completions.create(**kwargs)
completions_with_backoff(model="gpt-4o-mini", messages=[{"role": "user", "content": "Once upon a time,"}])
ChatCompletion(id='chatcmpl-8PAxGvV3GbLpnOoKSvJ00XCUdOglM', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content="in a faraway kingdom, there lived a young princess named Aurora. She was known for her beauty, grace, and kind heart. Aurora's kingdom was filled with lush green meadows, towering mountains, and sparkling rivers. The princess loved spending time exploring the enchanting forests surrounding her castle.\n\nOne day, while Aurora was wandering through the woods, she stumbled upon a hidden clearing. At the center stood a majestic oak tree, its branches reaching towards the sky. Aurora approached the tree with curiosity, and as she got closer, she noticed a small door at its base.\n\nIntrigued, she gently pushed open the door and was amazed to find herself in a magical realm. The forest transformed into a breathtaking wonderland, with colorful flowers blooming in every direction and woodland creatures frolicking joyously. Aurora's eyes widened with wonder as she explored this extraordinary world.\n\nAs she explored further, Aurora came across a small cottage in the distance. Curiosity overcame her, and she cautiously approached the cottage. To her surprise, an elderly woman with twinkling eyes and a warm smile stood in the doorway, welcoming her inside.\n\nThe woman revealed herself to be a fairy named Luna. Luna informed Aurora that she had been chosen to undertake a quest that would bring harmony to both her kingdom and the mystical realm. Aurora, eager to help, listened intently as Luna explained that a powerful enchantress had cast a spell on the kingdom, causing darkness and despair to loom over the land.\n\nTo break the curse, Aurora had to embark on a journey to retrieve a magical crystal hidden deep within the heart of an ancient cave. Without hesitation, the princess agreed and bid farewell to Luna, promising to return victorious.\n\nWith newfound determination, Aurora set off on her quest. Along the way, she encountered numerous challenges and obstacles but never lost hope. She often drew strength from the enchanting woodland creatures who accompanied her on this journey, reminding her that she was not alone.\n\nAfter a long and arduous journey, Aurora reached the entrance of the ancient cave. Inside, she faced a series of tests that pushed her physical and emotional limits. With sheer determination and unwavering courage, she overcame each trial, paving her way to the crystal's resting place.\n\nAs Aurora held the crystal in her hands, its warmth spread through her body. The artifact contained unimaginable power that could shatter the enchantress's curse and restore light to her kingdom. Brimming with joy and newfound strength, she made her way back to Luna's cottage.\n\nUpon her return, Aurora and Luna performed a powerful ritual, using the crystal's magic to break the curse. Waves of light and color spread across the kingdom, banishing darkness and despair. The once-gray skies turned blue, and laughter filled the air once again. The kingdom rejoiced, thanking Princess Aurora for her bravery and selflessness.\n\nFrom that day forward, Aurora was hailed as a hero, not only in her kingdom but also in the mystical realm. She continued to be a beacon of hope and kindness, reminding everyone that true courage lies within, waiting to be awakened.\n\nAnd so, Princess Aurora's tale lived on as a timeless reminder that even in the darkest of times, there is always light and hope to be found.", role='assistant', function_call=None, tool_calls=None)], created=1701011002, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=657, prompt_tokens=12, total_tokens=669))
退避到另一个模型
如果您在主模型上遇到速率限制错误，一种选择是切换到辅助模型。此方法有助于在主模型被限制或不可用时保持应用程序的响应能力。
但是，备用模型的准确性、延迟和成本可能差异很大。因此，此策略可能不适用于所有用例；特别是那些需要高度一致结果的用例。此外，请记住，某些模型共享速率限制，这可能会降低简单切换模型的有效性。您可以在组织限制页面上查看共享限制的模型。
在将此方法部署到生产环境之前，请彻底测试它对输出质量、用户体验和运营预算的影响。使用相关评估验证您的备用解决方案，以确保它满足您的要求并在实际条件下保持可接受的性能。
def completions_with_fallback(fallback_model, **kwargs):
try:
return client.chat.completions.create(**kwargs)
except openai.RateLimitError:
kwargs['model'] = fallback_model
return client.chat.completions.create(**kwargs)
completions_with_fallback(fallback_model="gpt-4o", model="gpt-4o-mini", messages=[{"role": "user", "content": "Once upon a time,"}])
ChatCompletion(id='chatcmpl-AsX9Zts2toXoKA80ZujeWMXKMolBy', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named Elara. Elara was known for her adventurous spirit and her unwavering curiosity about the world beyond her village. She often spent her days wandering the meadows, exploring the enchanted forest, and collecting wildflowers.\n\nOne sunny afternoon, while she was picking daisies near the edge of the forest, Elara stumbled upon an old, ornate key half-buried in the ground. Intrigued, she dusted it off and inspected it closely. The key was beautifully crafted, with intricate patterns carved into its metal. Elara felt a strange pull towards it, as if it were meant for her.\n\nDetermined to uncover its secrets, Elara ran back to the village, her heart racing with excitement. She gathered her closest friends—Jasper, a clever boy with a knack for puzzles, and Lila, a brave girl who loved to climb trees—and shared her discovery with them.\n\n"Do you think it belongs to a hidden treasure?" Jasper wondered, his eyes sparkling with mischief.\n\n"Or perhaps a secret door!" Lila added, her imagination running wild.\n\nTogether, they decided to seek out the source of the key. They combed through old tales told by the village elders, searching for any clues about a hidden door or treasure nearby. After days of excitement and exploration, they stumbled upon an ancient map tucked away in an old library. The map illustrated a long-lost castle deep within the enchanted forest, rumored to have been abandoned for centuries.\n\nWith the map in hand and their imaginations ignited, Elara, Jasper, and Lila set off towards the castle. The journey through the enchanted forest was filled with wonders—glowing fireflies, singing birds, and trees that seemed to whisper secrets as the wind rustled through their leaves. Eventually, they reached the castle, its crumbling walls draped in vines and mysterious shadows.\n\nStanding before the grand entrance, Elara held the key tightly in her hand. "This is it," she whispered, her heart pounding in anticipation. The friends exchanged nervous glances but shared the thrill of adventure. Together, they pushed open the heavy door, which creaked eerily as it swung wide.\n\nInside, they found a majestic hall adorned with fading tapestries and dust-laden chandeliers. In the center of the room stood a locked chest, adorned with the same intricate patterns as the key. Elara knelt beside it, her friends gathering around as she inserted the key into the lock. With a satisfying click, the chest opened to reveal a trove of shimmering jewels, ancient scrolls, and forgotten treasures.\n\nBut among the riches, they discovered something even more valuable—an old book filled with stories of bravery, friendship, and magic. As they turned the pages, each story seemed to echo their own journey and the spirit of adventure that had led them to this moment.\n\nElara, Jasper, and Lila realized that the true treasure was not the jewels or gold, but the experiences they had shared and the bond they had formed through their journey. They decided to take the book back to their village and share its tales with everyone, inspiring others to seek their own adventures and explore the wonders of the world around them.\n\nFrom that day forward, the trio became known as the Keepers of the Forest, guardians of the stories that connected their village to the magic of the enchanted world. And as they continued their adventures, they learned that the real magic lay within their hearts and the friendships they cherished. \n\nAnd so, they lived happily ever after, their spirits forever intertwined in a tapestry of tales waiting to be told.', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 774, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 775, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 747, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.08919167518615723, 'batcher_initial_queue_time': 0.008681058883666992}])], created=1737560517, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_72ed7ab54c', usage=CompletionUsage(completion_tokens=747, prompt_tokens=12, total_tokens=759, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))
将 max_tokens 减少到与预期补全匹配
速率限制使用量根据以下两者中的较大者计算：
max_tokens- 响应中允许的最大令牌数。- 输入中估计的令牌数 - 源自提示的字符数。
如果将 max_tokens 设置得太高，即使实际响应短得多，您的使用量也可能被高估。为避免过早达到速率限制，请将 max_tokens 配置为与其预期的响应大小紧密匹配。这可确保更准确的使用量计算，并有助于防止意外的限制。
def completions_with_max_tokens(**kwargs):
return client.chat.completions.create(**kwargs)
completions_with_max_tokens(model="gpt-4o-mini", messages=[{"role": "user", "content": "Once upon a time,"}], max_tokens=100)
ChatCompletion(id='chatcmpl-Aq0JmjugPw2i232ZEZuK5inHnx6Vc', choices=[Choice(finish_reason='length', index=0, logprobs=None, message=ChatCompletionMessage(content='in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named Lila. Lila was known for her boundless curiosity and adventurous spirit. She had a wild imagination, often spinning tales about the mysteries that lay beyond the village borders.\n\nOne day, while exploring the forest, Lila stumbled upon a hidden path she had never seen before. The path was winding and overgrown, beckoning her with whispers of adventure. Against her better judgment, she decided to', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None), internal_metrics=[{'cached_prompt_tokens': 0, 'total_accepted_tokens': 0, 'total_batched_tokens': 127, 'total_predicted_tokens': 0, 'total_rejected_tokens': 0, 'total_tokens_in_completion': 128, 'cached_embeddings_bytes': 0, 'cached_embeddings_n': 0, 'uncached_embeddings_bytes': 0, 'uncached_embeddings_n': 0, 'fetched_embeddings_bytes': 0, 'fetched_embeddings_n': 0, 'n_evictions': 0, 'sampling_steps': 100, 'sampling_steps_with_predictions': 0, 'batcher_ttft': 0.030033111572265625, 'batcher_initial_queue_time': 0.0006170272827148438}])], created=1736957642, model='gpt-4o-mini-2024-07-18', object='chat.completion', service_tier='default', system_fingerprint='fp_bd83329f63', usage=CompletionUsage(completion_tokens=100, prompt_tokens=12, total_tokens=112, completion_tokens_details=CompletionTokensDetails(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0), prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0, cached_tokens_internal=0)))
如何最大化给定速率限制的批量处理吞吐量
如果您正在处理来自用户的实时请求，退避和重试是最小化延迟同时避免速率限制错误的好策略。
但是，如果您正在处理大量批量数据，其中吞吐量比延迟更重要，那么除了退避和重试之外，您还可以做一些其他事情。
主动在请求之间添加延迟
如果您不断遇到速率限制，然后退避，然后再次遇到速率限制，然后再次退避，那么您的请求预算的很大一部分可能会在需要重试的请求上“浪费”。这会限制您的处理吞吐量，给定固定的速率限制。
在这里，一个潜在的解决方案是计算您的速率限制，并添加等于其倒数的延迟（例如，如果您的速率限制是每分钟 20 个请求，则在每个请求中添加 3-6 秒的延迟）。这可以帮助您在接近速率限制上限的情况下运行，而不会遇到它并产生浪费的请求。
添加延迟到请求的示例
import time
# 定义一个为 Completion API 调用添加延迟的函数
def delayed_completion(delay_in_seconds: float = 1, **kwargs):
"""延迟补全指定的时间。"""
# 睡眠延迟时间
time.sleep(delay_in_seconds)
# 调用 Completion API 并返回结果
return client.chat.completions.create(**kwargs)
# 根据您的速率限制计算延迟
rate_limit_per_minute = 20
delay = 60.0 / rate_limit_per_minute
delayed_completion(
delay_in_seconds=delay,
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Once upon a time,"}]
)
ChatCompletion(id='chatcmpl-8PAyCR1axKsomV0e349XiCN1Z81pH', choices=[Choice(finish_reason='stop', index=0, message=ChatCompletionMessage(content="in a small village, there lived a young girl named Maya. Maya was known for her kindness and love for nature. She spent hours exploring the forests surrounding the village, admiring the vibrant flowers and talking to the animals.\n\nOne sunny day, as Maya was picking wildflowers, she stumbled upon a wounded blackbird with a broken wing. Feeling sorry for the bird, Maya gently picked it up and cradled it in her hands. She knew she had to help the bird, so she hurried back to her cottage.\n\nMaya set up a cozy nest for the blackbird and carefully splinted its wing. She fed it worms and berries, doing everything she could to nurse it back to health. Each day, she would sing lullabies and tell stories to keep the blackbird company. Slowly, the bird's wing healed, and before long, it was ready to fly again.\n\nOn a beautiful morning, Maya opened the window of her cottage and released the blackbird into the sky. As the bird soared into the air, Maya's heart filled with joy and gratitude. Little did she know, this act of kindness would change her life forever.\n\nThe following night, a mysterious glowing light illuminated Maya's room. Startled, she sat up and saw a magical creature standing before her. It was a fairy, tiny yet radiating warmth and light.\n\nThe fairy introduced herself as Luna, the Guardian of the Forest. She had witnessed Maya's kindness towards the blackbird and had been watching her ever since. Luna explained that she had come to reward Maya for her selflessness.\n\nWith a wave of her wand, Luna granted Maya the ability to communicate with animals. Maya's eyes widened with amazement as she realized she could now understand the language of nature. Birds chirped melodies, rabbits whispered secrets, and trees shared their ancient wisdom.\n\nOver time, Maya's ability made her beloved by both humans and animals. Farmers sought her advice on how to care for their crops, and children flocked to her for stories of her enchanting encounters with the forest creatures. Maya used her gift to teach others about the importance of living in harmony with nature.\n\nAs years passed, Maya became known as the Village Guardian. She dedicated herself to protecting the surrounding forests from harm and educating others on sustainable living. The village flourished under Maya's guidance, and animals and humans lived side by side peacefully.\n\nAnd so, Maya's story became a legend passed down through generations. Her kindness, love for nature, and her ability to communicate with animals inspired people to treat the world around them with compassion and care.", role='assistant', function_call=None, tool_calls=None)], created=1701011060, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=524, prompt_tokens=12, total_tokens=536))
批处理请求
OpenAI API 对每分钟/每天的请求数（RPM/RPD）和每分钟的令牌数（TPM）强制执行单独的限制。如果您达到了 RPM 限制但仍有可用的 TPM 容量，请考虑将多个任务批处理到每个请求中。
通过将多个提示捆绑在一起，您可以减少每分钟发送的请求总数，这有助于避免达到 RPM 上限。如果您仔细管理 TPM 使用情况，此方法还可以带来更高的整体吞吐量。但是，请牢记以下几点：
- 每个模型在一次请求中可以处理的最大令牌数是有限制的。如果您的批量提示超过此限制，请求将失败或被截断。
- 如果任务延迟到分组到一个请求中，批处理可能会引入额外的等待时间。这可能会影响时间敏感型应用程序的用户体验。
- 发送多个提示时，响应对象可能不会以与提交的提示相同的顺序或格式返回。您应该通过后处理输出来尝试将每个响应与其对应的提示进行匹配。
无批处理示例
num_stories = 10
content = "Once upon a time,"
# 串行示例，每个请求一个故事补全
for _ in range(num_stories):
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": content}],
max_tokens=20,
)
print(content + response.choices[0].message.content)
Once upon a time,in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named
Once upon a time,Once upon a time, in a tranquil village nestled between rolling hills and lush forests, there lived a
Once upon a time,in a lush, green valley surrounded by towering mountains, there lay a small village called Eldergrove
Once upon a time,in a quaint little village nestled between rolling hills and a sparkling river, there lived a young girl named
Once upon a time,in a small village nestled between whispering woods and a sparkling river, there lived a curious young girl
Once upon a time,in a small village nestled between a vast forest and a shimmering lake, there lived a kind-hearted girl
Once upon a time,in a quaint little village nestled between rolling hills and a shimmering lake, there lived a curious girl named
Once upon a time,in a quaint little village nestled between emerald hills and a sparkling brook, there was a curious child named
Once upon a time,in a quaint little village nestled between rolling hills and lush forests, there lived an old clockmaker named
Once upon a time,in a quaint little village nestled between rolling hills and a shimmering lake, there lived a curious young girl
使用结构化输出在单个请求中批处理多个提示的示例
OpenAI 的结构化输出功能提供了一种健壮的方法，可以在单个请求中批处理多个提示。
在这里，您无需解析原始文本或期望模型遵循非正式格式，而是指定严格的模式。这可确保您的应用程序能够通过检查定义的结构来可靠地解析结果。这消除了对大量验证或复杂解析逻辑的需求，因为结构化输出保证了数据的一致性和类型安全。
from pydantic import BaseModel
# 定义结构化输出的 Pydantic 模型
class StoryResponse(BaseModel):
stories: list[str]
story_count: int
num_stories = 10
content = "Once upon a time,"
prompt_lines = [f"Story #{i+1}: {content}" for i in range(num_stories)]
prompt_text = "\n".join(prompt_lines)
messages = [
{
"role": "developer",
"content": "您是一位乐于助人的助手。请将每个提示作为单独的短故事进行回复。"
},
{
"role": "user",
"content": prompt_text
}
]
# 批量示例，在一个请求中包含所有故事补全并使用结构化输出
response = client.beta.chat.completions.parse(
model="gpt-4o-mini",
messages=messages,
response_format=StoryResponse,
)
print(response.choices[0].message.content)
{"stories":["Once upon a time, in a lush green valley, there lived a curious little fox named Felix. Every day, he would explore the woods, finding hidden glades and sparkling streams. One day, while chasing a butterfly, he stumbled upon a magical oak tree that granted wishes. Felix wished for courage and became the bravest fox in the land, helping his friends as they faced challenges together.","Once upon a time, a village known for its beautiful gardens fell into despair when a drought struck. The villagers prayed for rain but to no avail. One evening, a wise old woman arrived and told them of a hidden spring deep in the forest. With hope, the villagers embarked on a quest to find it, learning the value of teamwork and perseverance. Eventually, they found the spring, and the rain returned, reviving their gardens and spirits.","Once upon a time, in a kingdom high atop the clouds, lived Princess Lumina who had the ability to control the stars. But she felt lonely and longed for a companion. One night, she captured a shooting star and transformed it into a dashing young man named Orion. Together, they painted the night skies with adventures until Lumina learned to find joy in her own light.","Once upon a time, in a bustling bazaar in the heart of the city, there lived a clever merchant named Amina. She had a special talent for selling spices that made people fall in love. One day, a mysterious stranger entered her shop and bought a rare spice, causing an unexpected romance between two feuding families. Amina realized her spices held the power of unity, and she continued to spread love through her trade.","Once upon a time, a little turtle named Tilly dreamed of flying. Every day she watched the birds soar above her, wishing she could join them. One night, she met an old owl who shared stories of how to fly in one's heart, rather than with wings. Inspired, Tilly began to paint her dreams on shells, and soon, her colorful art attracted the birds. They carried her art into the sky, proving that dreams can take flight in unexpected ways.","Once upon a time, there was a forgotten castle hidden deep in the mountains. In this castle lived an ancient dragon named Ignis, who guarded a treasure of wisdom unlike any other. One day, a brave yet naive knight named Roland attempted to seize the treasure. But Ignis offered him a riddle instead. After solving it, Roland realized that the true treasure was knowledge and understanding. He left the castle as a wiser man, sharing Ignis's teachings with his kingdom.","Once upon a time, in a world where colors had feelings, there lived a dull gray town where nobody smiled. One day, a little girl named Bloom arrived, carrying a bright yellow paintbrush. She began to paint laughter and joy on the walls. Slowly, the townspeople found happiness in her colors and learned to express their emotions. Eventually, the town transformed into a vibrant place where every day was a celebration of life.","Once upon a time, an old clockmaker named Mr. Tick was known for creating the finest clocks in the town. But his favorite creation was an enchanted clock that could tell stories of the past. One day, a little girl named Clara stumbled into his shop and begged him to tell her a story. Mr. Tick set the clock and took her on a journey through time, where Clara learned the importance of history and family. Inspired, she decided to become a storyteller.","Once upon a time, in a small fishing village, a mysterious blue whale appeared off the coast every summer. Legend had it that the whale could grant one wish to the person who dared to swim alongside it. A daring young boy named Leo decided to brave the waters. As he swam next to the majestic creature, he wished for prosperity for his village. From that day onward, the village thrived, and they celebrated the bond of friendship with the whale every summer.","Once upon a time, in a land of giants, there lived a tiny girl named Fiona. Despite her size, she had a heart full of ambition. She dreamt of building a bridge between her village and the giants’ realm to facilitate friendship. With determination and ingenuity, she crafted a plan. When the giants saw her efforts, they helped her, and together they constructed a magnificent bridge. Fiona's courage became a legend, and the two realms flourished in harmony."],"story_count":10}
示例并行处理脚本
我们编写了一个用于并行处理大量 API 请求的示例脚本：api_request_parallel_processor.py。
该脚本结合了一些便捷功能：
- 从文件流式传输请求，以避免内存不足的大型作业
- 同时发出请求，以最大化吞吐量
- 限制请求和令牌使用量，以保持在速率限制内
- 重试失败的请求，以避免丢失数据
- 记录错误，以诊断请求问题
您可以随意按原样使用它，或根据您的需求进行修改。