Upload 225 files

.gitattributes

@@ -0,0 +1,2 @@
+docs/my-website/img/alt_dashboard.png filter=lfs diff=lfs merge=lfs -text
+docs/my-website/img/dashboard_log.png filter=lfs diff=lfs merge=lfs -text

docs/my-website/.gitignore

@@ -0,0 +1,20 @@
+# Dependencies
+/node_modules
+
+# Production
+/build
+
+# Generated files
+.docusaurus
+.cache-loader
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

docs/my-website/Dockerfile

@@ -0,0 +1,9 @@
+FROM python:3.10
+
+COPY . /app
+WORKDIR /app
+RUN pip install -r requirements.txt
+
+EXPOSE $PORT 
+
+CMD litellm --host 0.0.0.0 --port $PORT --workers 10 --config config.yaml

docs/my-website/README.md

@@ -0,0 +1,41 @@
+# Website
+
+This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
+
+### Installation
+
+```
+$ yarn
+```
+
+### Local Development
+
+```
+$ yarn start
+```
+
+This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
+
+### Build
+
+```
+$ yarn build
+```
+
+This command generates static content into the `build` directory and can be served using any static contents hosting service.
+
+### Deployment
+
+Using SSH:
+
+```
+$ USE_SSH=true yarn deploy
+```
+
+Not using SSH:
+
+```
+$ GIT_USER=<Your GitHub username> yarn deploy
+```
+
+If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the `gh-pages` branch.

docs/my-website/babel.config.js

@@ -0,0 +1,3 @@
+module.exports = {
+  presets: [require.resolve('@docusaurus/core/lib/babel/preset')],
+};

docs/my-website/blog/2021-08-26-welcome/index.md

@@ -0,0 +1,43 @@
+# 🚅 litellm
+A light 100 line package to simplify calling OpenAI, Azure, Cohere, Anthropic APIs 
+
+###### litellm manages:
+* Calling all LLM APIs using the OpenAI format - `completion(model, messages)`
+* Consistent output for all LLM APIs, text responses will always be available at `['choices'][0]['message']['content']`
+* Consistent Exceptions for all LLM APIs, we map RateLimit, Context Window, and Authentication Error exceptions across all providers to their OpenAI equivalents. [see Code](https://github.com/BerriAI/litellm/blob/ba1079ff6698ef238c5c7f771dd2b698ec76f8d9/litellm/utils.py#L250)
+
+###### observability:
+* Logging - see exactly what the raw model request/response is by plugging in your own function `completion(.., logger_fn=your_logging_fn)` and/or print statements from the package `litellm.set_verbose=True`
+* Callbacks - automatically send your data to Helicone, Sentry, Posthog, Slack - `litellm.success_callbacks`, `litellm.failure_callbacks` [see Callbacks](https://litellm.readthedocs.io/en/latest/advanced/)
+
+## Quick Start
+Go directly to code: [Getting Started Notebook](https://colab.research.google.com/drive/1gR3pY-JzDZahzpVdbGBtrNGDBmzUNJaJ?usp=sharing)
+### Installation
+```
+pip install litellm
+```
+
+### Usage
+```python
+from litellm import completion
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "openai key"
+os.environ["COHERE_API_KEY"] = "cohere key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+
+# cohere call
+response = completion("command-nightly", messages)
+```
+Need Help / Support : [see troubleshooting](https://litellm.readthedocs.io/en/latest/troubleshoot)
+
+## Why did we build liteLLM 
+- **Need for simplicity**: Our code started to get extremely complicated managing & translating calls between Azure, OpenAI, Cohere
+
+## Support
+* [Meet with us 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+* Contact us at [email protected] / [email protected]

docs/my-website/docs/budget_manager.md

@@ -0,0 +1,248 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Budget Manager
+
+Don't want to get crazy bills because either while you're calling LLM APIs **or** while your users are calling them? use this. 
+
+LiteLLM exposes: 
+* `litellm.max_budget`: a global variable you can use to set the max budget (in USD) across all your litellm calls. If this budget is exceeded, it will raise a BudgetExceededError 
+* `BudgetManager`: A class to help set budgets per user. BudgetManager creates a dictionary to manage the user budgets, where the key is user and the object is their current cost + model-specific costs. 
+
+## quick start
+
+```python 
+import litellm, os 
+from litellm import completion
+
+# set env variable 
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+litellm.max_budget = 0.001 # sets a max budget of $0.001
+
+messages = [{"role": "user", "content": "Hey, how's it going"}]
+completion(model="gpt-4", messages=messages)
+print(litellm._current_cost)
+completion(model="gpt-4", messages=messages)
+```
+
+## User-based rate limiting 
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/LiteLLM_User_Based_Rate_Limits.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+```python
+from litellm import BudgetManager, completion 
+
+budget_manager = BudgetManager(project_name="test_project")
+
+user = "1234"
+
+# create a budget if new user user
+if not budget_manager.is_valid_user(user):
+    budget_manager.create_budget(total_budget=10, user=user)
+
+# check if a given call can be made
+if budget_manager.get_current_cost(user=user) <= budget_manager.get_total_budget(user):
+    response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])
+    budget_manager.update_cost(completion_obj=response, user=user)
+else:
+    response = "Sorry - no budget!"
+```
+
+[**Implementation Code**](https://github.com/BerriAI/litellm/blob/main/litellm/budget_manager.py)
+
+## use with Text Input / Output
+
+Update cost by just passing in the text input / output and model name. 
+
+```python
+from litellm import BudgetManager
+
+budget_manager = BudgetManager(project_name="test_project")
+user = "12345"
+budget_manager.create_budget(total_budget=10, user=user, duration="daily")
+
+input_text = "hello world"
+output_text = "it's a sunny day in san francisco"
+model = "gpt-3.5-turbo"
+
+budget_manager.update_cost(user=user, model=model, input_text=input_text, output_text=output_text) # 👈
+print(budget_manager.get_current_cost(user))
+```
+
+## advanced usage
+In production, we will need to 
+* store user budgets in a database
+* reset user budgets based on a set duration 
+
+
+
+### LiteLLM API
+
+The LiteLLM API provides both. It stores the user object in a hosted db, and runs a cron job daily to reset user-budgets based on the set duration (e.g. reset budget daily/weekly/monthly/etc.). 
+
+**Usage**
+```python
+budget_manager = BudgetManager(project_name="<my-unique-project>", client_type="hosted")
+```
+
+**Complete Code**
+```python
+from litellm import BudgetManager, completion 
+
+budget_manager = BudgetManager(project_name="<my-unique-project>", client_type="hosted")
+
+user = "1234"
+
+# create a budget if new user user
+if not budget_manager.is_valid_user(user):
+    budget_manager.create_budget(total_budget=10, user=user, duration="monthly") # 👈 duration = 'daily'/'weekly'/'monthly'/'yearly'
+
+# check if a given call can be made
+if budget_manager.get_current_cost(user=user) <= budget_manager.get_total_budget(user):
+    response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])
+    budget_manager.update_cost(completion_obj=response, user=user)
+else:
+    response = "Sorry - no budget!"
+```
+
+### Self-hosted
+
+To use your own db, set the BudgetManager client type to `hosted` **and** set the api_base. 
+
+Your api is expected to expose `/get_budget` and `/set_budget` endpoints. [See code for details](https://github.com/BerriAI/litellm/blob/27f1051792176a7eb1fe3b72b72bccd6378d24e9/litellm/budget_manager.py#L7)
+
+**Usage**
+```python
+budget_manager = BudgetManager(project_name="<my-unique-project>", client_type="hosted", api_base="your_custom_api")
+```
+**Complete Code**
+```python
+from litellm import BudgetManager, completion 
+
+budget_manager = BudgetManager(project_name="<my-unique-project>", client_type="hosted", api_base="your_custom_api")
+
+user = "1234"
+
+# create a budget if new user user
+if not budget_manager.is_valid_user(user):
+    budget_manager.create_budget(total_budget=10, user=user, duration="monthly") # 👈 duration = 'daily'/'weekly'/'monthly'/'yearly'
+
+# check if a given call can be made
+if budget_manager.get_current_cost(user=user) <= budget_manager.get_total_budget(user):
+    response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])
+    budget_manager.update_cost(completion_obj=response, user=user)
+else:
+    response = "Sorry - no budget!"
+```
+
+## Budget Manager Class
+The `BudgetManager` class is used to manage budgets for different users. It provides various functions to create, update, and retrieve budget information. 
+
+Below is a list of public functions exposed by the Budget Manager class and their input/outputs. 
+
+### __init__
+```python
+def __init__(self, project_name: str, client_type: str = "local", api_base: Optional[str] = None)
+```
+- `project_name` (str): The name of the project.
+- `client_type` (str): The client type ("local" or "hosted"). Defaults to "local".
+- `api_base` (Optional[str]): The base URL of the API. Defaults to None.
+
+
+### create_budget
+```python
+def create_budget(self, total_budget: float, user: str, duration: Literal["daily", "weekly", "monthly", "yearly"], created_at: float = time.time())
+```
+Creates a budget for a user.
+
+- `total_budget` (float): The total budget of the user.
+- `user` (str): The user id.
+- `duration` (Literal["daily", "weekly", "monthly", "yearly"]): The budget duration.
+- `created_at` (float): The creation time. Default is the current time.
+
+### projected_cost
+```python
+def projected_cost(self, model: str, messages: list, user: str)
+```
+Computes the projected cost for a session.
+
+- `model` (str): The name of the model.
+- `messages` (list): The list of messages.
+- `user` (str): The user id.
+
+### get_total_budget
+```python
+def get_total_budget(self, user: str)
+```
+Returns the total budget of a user.
+
+- `user` (str): user id.
+
+### update_cost
+```python
+def update_cost(self, completion_obj: ModelResponse, user: str)
+```
+Updates the user's cost.
+
+- `completion_obj` (ModelResponse): The completion object received from the model.
+- `user` (str): The user id.
+
+### get_current_cost
+```python
+def get_current_cost(self, user: str)
+```
+Returns the current cost of a user.
+
+- `user` (str): The user id.
+
+### get_model_cost 
+```python
+def get_model_cost(self, user: str)
+```
+Returns the model cost of a user.
+
+- `user` (str): The user id.
+
+### is_valid_user 
+```python
+def is_valid_user(self, user: str) -> bool
+```
+Checks if a user is valid.
+
+- `user` (str): The user id.
+
+### get_users 
+```python
+def get_users(self)
+```
+Returns a list of all users.
+
+### reset_cost 
+```python
+def reset_cost(self, user: str)
+```
+Resets the cost of a user.
+
+- `user` (str): The user id.
+
+### reset_on_duration 
+```python
+def reset_on_duration(self, user: str)
+```
+Resets the cost of a user based on the duration.
+
+- `user` (str): The user id.
+
+### update_budget_all_users 
+```python
+def update_budget_all_users(self)
+```
+Updates the budget for all users.
+
+### save_data 
+```python
+def save_data(self)
+```
+Stores the user dictionary.

docs/my-website/docs/caching/caching_api.md

@@ -0,0 +1,75 @@
+# Hosted Cache - api.litellm.ai
+
+Use api.litellm.ai for caching `completion()` and `embedding()` responses
+
+## Quick Start Usage - Completion
+```python
+import litellm
+from litellm import completion
+from litellm.caching import Cache
+litellm.cache = Cache(type="hosted") # init cache to use api.litellm.ai
+
+# Make completion calls
+response1 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}]
+    caching=True
+)
+
+response2 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}],
+    caching=True
+)
+# response1 == response2, response 1 is cached
+```
+
+
+## Usage - Embedding()
+
+```python
+import time
+import litellm
+from litellm import completion, embedding
+from litellm.caching import Cache
+litellm.cache = Cache(type="hosted")
+
+start_time = time.time()
+embedding1 = embedding(model="text-embedding-ada-002", input=["hello from litellm"*5], caching=True)
+end_time = time.time()
+print(f"Embedding 1 response time: {end_time - start_time} seconds")
+
+start_time = time.time()
+embedding2 = embedding(model="text-embedding-ada-002", input=["hello from litellm"*5], caching=True)
+end_time = time.time()
+print(f"Embedding 2 response time: {end_time - start_time} seconds")
+```
+
+## Caching with Streaming 
+LiteLLM can cache your streamed responses for you
+
+### Usage
+```python
+import litellm
+from litellm import completion
+from litellm.caching import Cache
+litellm.cache = Cache(type="hosted")
+
+# Make completion calls
+response1 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}], 
+    stream=True,
+    caching=True)
+for chunk in response1:
+    print(chunk)
+
+
+response2 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}], 
+    stream=True,
+    caching=True)
+for chunk in response2:
+    print(chunk)
+```

docs/my-website/docs/caching/local_caching.md

@@ -0,0 +1,92 @@
+# LiteLLM - Local Caching
+
+## Caching `completion()` and `embedding()` calls when switched on
+
+liteLLM implements exact match caching and supports the following Caching:
+* In-Memory Caching [Default]
+* Redis Caching Local
+* Redis Caching Hosted
+
+## Quick Start Usage - Completion
+Caching - cache
+Keys in the cache are `model`, the following example will lead to a cache hit
+```python
+import litellm
+from litellm import completion
+from litellm.caching import Cache
+litellm.cache = Cache()
+
+# Make completion calls
+response1 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}]
+    caching=True
+)
+response2 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}],
+    caching=True
+)
+
+# response1 == response2, response 1 is cached
+```
+
+## Custom Key-Value Pairs 
+Add custom key-value pairs to your cache. 
+
+```python 
+from litellm.caching import Cache
+cache = Cache()
+
+cache.add_cache(cache_key="test-key", result="1234")
+
+cache.get_cache(cache_key="test-key)
+```
+
+## Caching with Streaming 
+LiteLLM can cache your streamed responses for you
+
+### Usage
+```python
+import litellm
+from litellm import completion
+from litellm.caching import Cache
+litellm.cache = Cache()
+
+# Make completion calls
+response1 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}], 
+    stream=True,
+    caching=True)
+for chunk in response1:
+    print(chunk)
+response2 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}], 
+    stream=True,
+    caching=True)
+for chunk in response2:
+    print(chunk)
+```
+
+## Usage - Embedding()
+1. Caching - cache
+Keys in the cache are `model`, the following example will lead to a cache hit
+```python
+import time
+import litellm
+from litellm import embedding
+from litellm.caching import Cache
+litellm.cache = Cache()
+
+start_time = time.time()
+embedding1 = embedding(model="text-embedding-ada-002", input=["hello from litellm"*5], caching=True)
+end_time = time.time()
+print(f"Embedding 1 response time: {end_time - start_time} seconds")
+
+start_time = time.time()
+embedding2 = embedding(model="text-embedding-ada-002", input=["hello from litellm"*5], caching=True)
+end_time = time.time()
+print(f"Embedding 2 response time: {end_time - start_time} seconds")
+```

docs/my-website/docs/caching/redis_cache.md

@@ -0,0 +1,73 @@
+# Redis Cache
+### Pre-requisites
+Install redis
+```
+pip install redis
+```
+For the hosted version you can setup your own Redis DB here: https://app.redislabs.com/
+### Usage
+```python
+import litellm
+from litellm import completion
+from litellm.caching import Cache
+litellm.cache = Cache(type="redis", host=<host>, port=<port>, password=<password>)
+
+# Make completion calls
+response1 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}],
+    caching=True
+)
+response2 = completion(
+    model="gpt-3.5-turbo", 
+    messages=[{"role": "user", "content": "Tell me a joke."}],
+    caching=True
+)
+
+# response1 == response2, response 1 is cached
+```
+
+### Custom Cache Keys:
+
+Define function to return cache key
+```python
+# this function takes in *args, **kwargs and returns the key you want to use for caching
+def custom_get_cache_key(*args, **kwargs):
+    # return key to use for your cache:
+    key = kwargs.get("model", "") + str(kwargs.get("messages", "")) + str(kwargs.get("temperature", "")) + str(kwargs.get("logit_bias", ""))
+    print("key for cache", key)
+    return key
+
+```
+
+Set your function as litellm.cache.get_cache_key
+```python
+from litellm.caching import Cache
+
+cache = Cache(type="redis", host=os.environ['REDIS_HOST'], port=os.environ['REDIS_PORT'], password=os.environ['REDIS_PASSWORD'])
+
+cache.get_cache_key = custom_get_cache_key # set get_cache_key function for your cache
+
+litellm.cache = cache # set litellm.cache to your cache 
+
+```
+
+### Detecting Cached Responses
+For resposes that were returned as cache hit, the response includes a param `cache` = True 
+
+Example response with cache hit
+```python
+{
+    'cache': True,
+    'id': 'chatcmpl-7wggdzd6OXhgE2YhcLJHJNZsEWzZ2', 
+    'created': 1694221467, 
+    'model': 'gpt-3.5-turbo-0613', 
+    'choices': [
+        {
+            'index': 0, 'message': {'role': 'assistant', 'content': 'I\'m sorry, but I couldn\'t find any information about "litellm" or how many stars it has. It is possible that you may be referring to a specific product, service, or platform that I am not familiar with. Can you please provide more context or clarify your question?'
+        }, 'finish_reason': 'stop'}
+    ], 
+    'usage': {'prompt_tokens': 17, 'completion_tokens': 59, 'total_tokens': 76}, 
+}
+
+```

docs/my-website/docs/completion/batching.md

@@ -0,0 +1,182 @@
+# Batching Completion()
+LiteLLM allows you to:
+* Send many completion calls to 1 model
+* Send 1 completion call to many models: Return Fastest Response
+* Send 1 completion call to many models: Return All Responses
+
+## Send multiple completion calls to 1 model
+
+In the batch_completion method, you provide a list of `messages` where each sub-list of messages is passed to `litellm.completion()`, allowing you to process multiple prompts efficiently in a single API call.
+
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/LiteLLM_batch_completion.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+### Example Code
+```python
+import litellm
+import os
+from litellm import batch_completion
+
+os.environ['ANTHROPIC_API_KEY'] = ""
+
+
+responses = batch_completion(
+    model="claude-2",
+    messages = [
+        [
+            {
+                "role": "user",
+                "content": "good morning? "
+            }
+        ],
+        [
+            {
+                "role": "user",
+                "content": "what's the time? "
+            }
+        ]
+    ]
+)
+```
+
+## Send 1 completion call to many models: Return Fastest Response
+This makes parallel calls to the specified `models` and returns the first response 
+
+Use this to reduce latency
+
+### Example Code
+```python
+import litellm
+import os
+from litellm import batch_completion_models
+
+os.environ['ANTHROPIC_API_KEY'] = ""
+os.environ['OPENAI_API_KEY'] = ""
+os.environ['COHERE_API_KEY'] = ""
+
+response = batch_completion_models(
+    models=["gpt-3.5-turbo", "claude-instant-1.2", "command-nightly"], 
+    messages=[{"role": "user", "content": "Hey, how's it going"}]
+)
+print(result)
+```
+
+### Output
+Returns the first response
+```json
+{
+  "object": "chat.completion",
+  "choices": [
+    {
+      "finish_reason": "stop",
+      "index": 0,
+      "message": {
+        "content": " I'm doing well, thanks for asking! I'm an AI assistant created by Anthropic to be helpful, harmless, and honest.",
+        "role": "assistant",
+        "logprobs": null
+      }
+    }
+  ],
+  "id": "chatcmpl-23273eed-e351-41be-a492-bafcf5cf3274",
+  "created": 1695154628.2076092,
+  "model": "command-nightly",
+  "usage": {
+    "prompt_tokens": 6,
+    "completion_tokens": 14,
+    "total_tokens": 20
+  }
+}
+```
+
+## Send 1 completion call to many models: Return All Responses
+This makes parallel calls to the specified models and returns all responses
+
+Use this to process requests concurrently and get responses from multiple models.
+
+### Example Code
+```python
+import litellm
+import os
+from litellm import batch_completion_models_all_responses
+
+os.environ['ANTHROPIC_API_KEY'] = ""
+os.environ['OPENAI_API_KEY'] = ""
+os.environ['COHERE_API_KEY'] = ""
+
+responses = batch_completion_models_all_responses(
+    models=["gpt-3.5-turbo", "claude-instant-1.2", "command-nightly"], 
+    messages=[{"role": "user", "content": "Hey, how's it going"}]
+)
+print(responses)
+
+```
+
+### Output
+
+```json
+[<ModelResponse chat.completion id=chatcmpl-e673ec8e-4e8f-4c9e-bf26-bf9fa7ee52b9 at 0x103a62160> JSON: {
+  "object": "chat.completion",
+  "choices": [
+    {
+      "finish_reason": "stop_sequence",
+      "index": 0,
+      "message": {
+        "content": " It's going well, thank you for asking! How about you?",
+        "role": "assistant",
+        "logprobs": null
+      }
+    }
+  ],
+  "id": "chatcmpl-e673ec8e-4e8f-4c9e-bf26-bf9fa7ee52b9",
+  "created": 1695222060.917964,
+  "model": "claude-instant-1.2",
+  "usage": {
+    "prompt_tokens": 14,
+    "completion_tokens": 9,
+    "total_tokens": 23
+  }
+}, <ModelResponse chat.completion id=chatcmpl-ab6c5bd3-b5d9-4711-9697-e28d9fb8a53c at 0x103a62b60> JSON: {
+  "object": "chat.completion",
+  "choices": [
+    {
+      "finish_reason": "stop",
+      "index": 0,
+      "message": {
+        "content": " It's going well, thank you for asking! How about you?",
+        "role": "assistant",
+        "logprobs": null
+      }
+    }
+  ],
+  "id": "chatcmpl-ab6c5bd3-b5d9-4711-9697-e28d9fb8a53c",
+  "created": 1695222061.0445492,
+  "model": "command-nightly",
+  "usage": {
+    "prompt_tokens": 6,
+    "completion_tokens": 14,
+    "total_tokens": 20
+  }
+}, <OpenAIObject chat.completion id=chatcmpl-80szFnKHzCxObW0RqCMw1hWW1Icrq at 0x102dd6430> JSON: {
+  "id": "chatcmpl-80szFnKHzCxObW0RqCMw1hWW1Icrq",
+  "object": "chat.completion",
+  "created": 1695222061,
+  "model": "gpt-3.5-turbo-0613",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "Hello! I'm an AI language model, so I don't have feelings, but I'm here to assist you with any questions or tasks you might have. How can I help you today?"
+      },
+      "finish_reason": "stop"
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 13,
+    "completion_tokens": 39,
+    "total_tokens": 52
+  }
+}]
+
+```

docs/my-website/docs/completion/config.md

@@ -0,0 +1,49 @@
+# Model Config
+
+Model-specific changes can make our code complicated, making it harder to debug errors. Use model configs to simplify this. 
+
+### usage
+
+Handling prompt logic. Different models have different context windows. Use `adapt_to_prompt_size` to select the right model for the prompt (in case the current model is too small).
+
+
+```python
+from litellm import completion_with_config 
+import os 
+
+config = {
+    "available_models": ["gpt-3.5-turbo", "claude-instant-1", "gpt-3.5-turbo-16k"],
+    "adapt_to_prompt_size": True, # 👈 key change
+}
+
+# set env var
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+os.environ["ANTHROPIC_API_KEY"] = "your-api-key"
+
+
+sample_text = "how does a court case get to the Supreme Court?" * 1000
+messages = [{"content": sample_text, "role": "user"}]
+response = completion_with_config(model="gpt-3.5-turbo", messages=messages, config=config)
+```
+
+[**See Code**](https://github.com/BerriAI/litellm/blob/30724d9e51cdc2c3e0eb063271b4f171bc01b382/litellm/utils.py#L2783)
+
+### Complete Config Structure
+
+```python
+config = {
+    "default_fallback_models": # [Optional] List of model names to try if a call fails
+    "available_models": # [Optional] List of all possible models you could call 
+    "adapt_to_prompt_size": # [Optional] True/False - if you want to select model based on prompt size (will pick from available_models)
+    "model": {
+        "model-name": {
+            "needs_moderation": # [Optional] True/False - if you want to call openai moderations endpoint before making completion call. Will raise exception, if flagged. 
+            "error_handling": {
+                "error-type": { # One of the errors listed here - https://docs.litellm.ai/docs/exception_mapping#custom-mapping-list
+                    "fallback_model": "" # str, name of the model it should try instead, when that error occurs 
+                }
+            }
+        }
+    }
+}
+```

docs/my-website/docs/completion/function_call.md

@@ -0,0 +1,545 @@
+# Function Calling 
+Function calling is supported with the following models on OpenAI, Azure OpenAI
+
+- gpt-4
+- gpt-4-1106-preview
+- gpt-4-0613
+- gpt-3.5-turbo
+- gpt-3.5-turbo-1106
+- gpt-3.5-turbo-0613
+- Non OpenAI LLMs (litellm adds the function call to the prompt for these llms)
+
+In addition, parallel function calls is supported on the following models:
+- gpt-4-1106-preview
+- gpt-3.5-turbo-1106
+
+## Parallel Function calling
+Parallel function calling is the model's ability to perform multiple function calls together, allowing the effects and results of these function calls to be resolved in parallel
+
+## Quick Start - gpt-3.5-turbo-1106
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/Parallel_function_calling.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+In this example we define a single function `get_current_weather`. 
+
+- Step 1: Send the model the `get_current_weather` with the user question
+- Step 2: Parse the output from the model response - Execute the `get_current_weather` with the model provided args
+- Step 3: Send the model the output from running the `get_current_weather` function
+
+
+### Full Code - Parallel function calling with `gpt-3.5-turbo-1106`
+
+```python
+import litellm
+import json
+# set openai api key
+import os
+os.environ['OPENAI_API_KEY'] = "" # litellm reads OPENAI_API_KEY from .env and sends the request
+
+# Example dummy function hard coded to return the same weather
+# In production, this could be your backend API or an external API
+def get_current_weather(location, unit="fahrenheit"):
+    """Get the current weather in a given location"""
+    if "tokyo" in location.lower():
+        return json.dumps({"location": "Tokyo", "temperature": "10", "unit": "celsius"})
+    elif "san francisco" in location.lower():
+        return json.dumps({"location": "San Francisco", "temperature": "72", "unit": "fahrenheit"})
+    elif "paris" in location.lower():
+        return json.dumps({"location": "Paris", "temperature": "22", "unit": "celsius"})
+    else:
+        return json.dumps({"location": location, "temperature": "unknown"})
+
+
+def test_parallel_function_call():
+    try:
+        # Step 1: send the conversation and available functions to the model
+        messages = [{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}]
+        tools = [
+            {
+                "type": "function",
+                "function": {
+                    "name": "get_current_weather",
+                    "description": "Get the current weather in a given location",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "location": {
+                                "type": "string",
+                                "description": "The city and state, e.g. San Francisco, CA",
+                            },
+                            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                        },
+                        "required": ["location"],
+                    },
+                },
+            }
+        ]
+        response = litellm.completion(
+            model="gpt-3.5-turbo-1106",
+            messages=messages,
+            tools=tools,
+            tool_choice="auto",  # auto is default, but we'll be explicit
+        )
+        print("\nFirst LLM Response:\n", response)
+        response_message = response.choices[0].message
+        tool_calls = response_message.tool_calls
+
+        print("\nLength of tool calls", len(tool_calls))
+
+        # Step 2: check if the model wanted to call a function
+        if tool_calls:
+            # Step 3: call the function
+            # Note: the JSON response may not always be valid; be sure to handle errors
+            available_functions = {
+                "get_current_weather": get_current_weather,
+            }  # only one function in this example, but you can have multiple
+            messages.append(response_message)  # extend conversation with assistant's reply
+
+            # Step 4: send the info for each function call and function response to the model
+            for tool_call in tool_calls:
+                function_name = tool_call.function.name
+                function_to_call = available_functions[function_name]
+                function_args = json.loads(tool_call.function.arguments)
+                function_response = function_to_call(
+                    location=function_args.get("location"),
+                    unit=function_args.get("unit"),
+                )
+                messages.append(
+                    {
+                        "tool_call_id": tool_call.id,
+                        "role": "tool",
+                        "name": function_name,
+                        "content": function_response,
+                    }
+                )  # extend conversation with function response
+            second_response = litellm.completion(
+                model="gpt-3.5-turbo-1106",
+                messages=messages,
+            )  # get a new response from the model where it can see the function response
+            print("\nSecond LLM response:\n", second_response)
+            return second_response
+    except Exception as e:
+      print(f"Error occurred: {e}")
+
+test_parallel_function_call()
+```
+
+### Explanation - Parallel function calling
+Below is an explanation of what is happening in the code snippet above for Parallel function calling with `gpt-3.5-turbo-1106`
+### Step1: litellm.completion() with `tools` set to `get_current_weather`
+```python
+import litellm
+import json
+# set openai api key
+import os
+os.environ['OPENAI_API_KEY'] = "" # litellm reads OPENAI_API_KEY from .env and sends the request
+# Example dummy function hard coded to return the same weather
+# In production, this could be your backend API or an external API
+def get_current_weather(location, unit="fahrenheit"):
+    """Get the current weather in a given location"""
+    if "tokyo" in location.lower():
+        return json.dumps({"location": "Tokyo", "temperature": "10", "unit": "celsius"})
+    elif "san francisco" in location.lower():
+        return json.dumps({"location": "San Francisco", "temperature": "72", "unit": "fahrenheit"})
+    elif "paris" in location.lower():
+        return json.dumps({"location": "Paris", "temperature": "22", "unit": "celsius"})
+    else:
+        return json.dumps({"location": location, "temperature": "unknown"})
+
+messages = [{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}]
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_current_weather",
+            "description": "Get the current weather in a given location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city and state, e.g. San Francisco, CA",
+                    },
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    }
+]
+
+response = litellm.completion(
+    model="gpt-3.5-turbo-1106",
+    messages=messages,
+    tools=tools,
+    tool_choice="auto",  # auto is default, but we'll be explicit
+)
+print("\nLLM Response1:\n", response)
+response_message = response.choices[0].message
+tool_calls = response.choices[0].message.tool_calls
+```
+
+##### Expected output
+In the output you can see the model calls the function multiple times - for San Francisco, Tokyo, Paris
+```json
+ModelResponse(
+  id='chatcmpl-8MHBKZ9t6bXuhBvUMzoKsfmmlv7xq', 
+  choices=[
+    Choices(finish_reason='tool_calls', 
+    index=0, 
+    message=Message(content=None, role='assistant', 
+      tool_calls=[
+        ChatCompletionMessageToolCall(id='call_DN6IiLULWZw7sobV6puCji1O', function=Function(arguments='{"location": "San Francisco", "unit": "celsius"}', name='get_current_weather'), type='function'), 
+
+        ChatCompletionMessageToolCall(id='call_ERm1JfYO9AFo2oEWRmWUd40c', function=Function(arguments='{"location": "Tokyo", "unit": "celsius"}', name='get_current_weather'), type='function'), 
+        
+        ChatCompletionMessageToolCall(id='call_2lvUVB1y4wKunSxTenR0zClP', function=Function(arguments='{"location": "Paris", "unit": "celsius"}', name='get_current_weather'), type='function')
+        ]))
+    ], 
+    created=1700319953, 
+    model='gpt-3.5-turbo-1106', 
+    object='chat.completion', 
+    system_fingerprint='fp_eeff13170a',
+    usage={'completion_tokens': 77, 'prompt_tokens': 88, 'total_tokens': 165}, 
+    _response_ms=1177.372
+)
+```
+
+### Step 2 -  Parse the Model Response and Execute Functions
+After sending the initial request, parse the model response to identify the function calls it wants to make. In this example, we expect three tool calls, each corresponding to a location (San Francisco, Tokyo, and Paris). 
+
+```python
+# Check if the model wants to call a function
+if tool_calls:
+    # Execute the functions and prepare responses
+    available_functions = {
+        "get_current_weather": get_current_weather,
+    }
+
+    messages.append(response_message)  # Extend conversation with assistant's reply
+
+    for tool_call in tool_calls:
+      print(f"\nExecuting tool call\n{tool_call}")
+      function_name = tool_call.function.name
+      function_to_call = available_functions[function_name]
+      function_args = json.loads(tool_call.function.arguments)
+      # calling the get_current_weather() function
+      function_response = function_to_call(
+          location=function_args.get("location"),
+          unit=function_args.get("unit"),
+      )
+      print(f"Result from tool call\n{function_response}\n")
+
+      # Extend conversation with function response
+      messages.append(
+          {
+              "tool_call_id": tool_call.id,
+              "role": "tool",
+              "name": function_name,
+              "content": function_response,
+          }
+      )
+
+```
+
+### Step 3 - Second litellm.completion() call 
+Once the functions are executed, send the model the information for each function call and its response. This allows the model to generate a new response considering the effects of the function calls.
+```python
+second_response = litellm.completion(
+    model="gpt-3.5-turbo-1106",
+    messages=messages,
+)
+print("Second Response\n", second_response)
+```
+
+#### Expected output
+```json
+ModelResponse(
+  id='chatcmpl-8MHBLh1ldADBP71OrifKap6YfAd4w', 
+  choices=[
+    Choices(finish_reason='stop', index=0, 
+    message=Message(content="The current weather in San Francisco is 72°F, in Tokyo it's 10°C, and in Paris it's 22°C.", role='assistant'))
+  ], 
+  created=1700319955, 
+  model='gpt-3.5-turbo-1106', 
+  object='chat.completion', 
+  system_fingerprint='fp_eeff13170a', 
+  usage={'completion_tokens': 28, 'prompt_tokens': 169, 'total_tokens': 197}, 
+  _response_ms=1032.431
+)
+```
+
+## Parallel Function Calling - Azure OpenAI
+```python
+# set Azure env variables
+import os
+os.environ['AZURE_API_KEY'] = "" # litellm reads AZURE_API_KEY from .env and sends the request
+os.environ['AZURE_API_BASE'] = "https://openai-gpt-4-test-v-1.openai.azure.com/"
+os.environ['AZURE_API_VERSION'] = "2023-07-01-preview"
+
+import litellm
+import json
+# Example dummy function hard coded to return the same weather
+# In production, this could be your backend API or an external API
+def get_current_weather(location, unit="fahrenheit"):
+    """Get the current weather in a given location"""
+    if "tokyo" in location.lower():
+        return json.dumps({"location": "Tokyo", "temperature": "10", "unit": "celsius"})
+    elif "san francisco" in location.lower():
+        return json.dumps({"location": "San Francisco", "temperature": "72", "unit": "fahrenheit"})
+    elif "paris" in location.lower():
+        return json.dumps({"location": "Paris", "temperature": "22", "unit": "celsius"})
+    else:
+        return json.dumps({"location": location, "temperature": "unknown"})
+
+## Step 1: send the conversation and available functions to the model
+messages = [{"role": "user", "content": "What's the weather like in San Francisco, Tokyo, and Paris?"}]
+tools = [
+    {
+        "type": "function",
+        "function": {
+            "name": "get_current_weather",
+            "description": "Get the current weather in a given location",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "location": {
+                        "type": "string",
+                        "description": "The city and state, e.g. San Francisco, CA",
+                    },
+                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
+                },
+                "required": ["location"],
+            },
+        },
+    }
+]
+
+response = litellm.completion(
+    model="azure/chatgpt-functioncalling", # model = azure/<your-azure-deployment-name>
+    messages=messages,
+    tools=tools,
+    tool_choice="auto",  # auto is default, but we'll be explicit
+)
+print("\nLLM Response1:\n", response)
+response_message = response.choices[0].message
+tool_calls = response.choices[0].message.tool_calls
+print("\nTool Choice:\n", tool_calls)
+
+## Step 2 - Parse the Model Response and Execute Functions
+# Check if the model wants to call a function
+if tool_calls:
+    # Execute the functions and prepare responses
+    available_functions = {
+        "get_current_weather": get_current_weather,
+    }
+
+    messages.append(response_message)  # Extend conversation with assistant's reply
+
+    for tool_call in tool_calls:
+      print(f"\nExecuting tool call\n{tool_call}")
+      function_name = tool_call.function.name
+      function_to_call = available_functions[function_name]
+      function_args = json.loads(tool_call.function.arguments)
+      # calling the get_current_weather() function
+      function_response = function_to_call(
+          location=function_args.get("location"),
+          unit=function_args.get("unit"),
+      )
+      print(f"Result from tool call\n{function_response}\n")
+
+      # Extend conversation with function response
+      messages.append(
+          {
+              "tool_call_id": tool_call.id,
+              "role": "tool",
+              "name": function_name,
+              "content": function_response,
+          }
+      )
+
+## Step 3 - Second litellm.completion() call
+second_response = litellm.completion(
+    model="azure/chatgpt-functioncalling",
+    messages=messages,
+)
+print("Second Response\n", second_response)
+print("Second Response Message\n", second_response.choices[0].message.content)
+
+```
+
+## Deprecated - Function Calling with `completion(functions=functions)`
+```python
+import os, litellm
+from litellm import completion
+
+os.environ['OPENAI_API_KEY'] = ""
+
+messages = [
+    {"role": "user", "content": "What is the weather like in Boston?"}
+]
+
+# python function that will get executed
+def get_current_weather(location):
+  if location == "Boston, MA":
+    return "The weather is 12F"
+
+# JSON Schema to pass to OpenAI
+functions = [
+    {
+      "name": "get_current_weather",
+      "description": "Get the current weather in a given location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state, e.g. San Francisco, CA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["celsius", "fahrenheit"]
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  ]
+
+response = completion(model="gpt-3.5-turbo-0613", messages=messages, functions=functions)
+print(response)
+```
+
+## litellm.function_to_dict - Convert Functions to dictionary for OpenAI function calling
+`function_to_dict` allows you to pass a function docstring and produce a dictionary usable for OpenAI function calling
+
+### Using `function_to_dict`
+1. Define your function `get_current_weather`
+2. Add a docstring to your function `get_current_weather`
+3. Pass the function to `litellm.utils.function_to_dict` to get the dictionary for OpenAI function calling
+
+```python
+# function with docstring
+def get_current_weather(location: str, unit: str):
+        """Get the current weather in a given location
+
+        Parameters
+        ----------
+        location : str
+            The city and state, e.g. San Francisco, CA
+        unit : {'celsius', 'fahrenheit'}
+            Temperature unit
+
+        Returns
+        -------
+        str
+            a sentence indicating the weather
+        """
+        if location == "Boston, MA":
+            return "The weather is 12F"
+
+# use litellm.utils.function_to_dict to convert function to dict
+function_json = litellm.utils.function_to_dict(get_current_weather)
+print(function_json)
+```
+
+#### Output from function_to_dict
+```json
+{
+    'name': 'get_current_weather', 
+    'description': 'Get the current weather in a given location', 
+    'parameters': {
+        'type': 'object', 
+        'properties': {
+            'location': {'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA'}, 
+            'unit': {'type': 'string', 'description': 'Temperature unit', 'enum': "['fahrenheit', 'celsius']"}
+        }, 
+        'required': ['location', 'unit']
+    }
+}
+```
+
+### Using function_to_dict with Function calling
+```python
+import os, litellm
+from litellm import completion
+
+os.environ['OPENAI_API_KEY'] = ""
+
+messages = [
+    {"role": "user", "content": "What is the weather like in Boston?"}
+]
+
+def get_current_weather(location: str, unit: str):
+    """Get the current weather in a given location
+
+    Parameters
+    ----------
+    location : str
+        The city and state, e.g. San Francisco, CA
+    unit : str {'celsius', 'fahrenheit'}
+        Temperature unit
+
+    Returns
+    -------
+    str
+        a sentence indicating the weather
+    """
+    if location == "Boston, MA":
+        return "The weather is 12F"
+
+functions = [litellm.utils.function_to_dict(get_current_weather)]
+
+response = completion(model="gpt-3.5-turbo-0613", messages=messages, functions=functions)
+print(response)
+```
+
+## Function calling for Non-OpenAI LLMs
+
+### Adding Function to prompt
+For Non OpenAI LLMs LiteLLM allows you to add the function to the prompt set: `litellm.add_function_to_prompt = True`
+
+#### Usage
+```python
+import os, litellm
+from litellm import completion
+
+# IMPORTANT - Set this to TRUE to add the function to the prompt for Non OpenAI LLMs
+litellm.add_function_to_prompt = True # set add_function_to_prompt for Non OpenAI LLMs
+
+os.environ['ANTHROPIC_API_KEY'] = ""
+
+messages = [
+    {"role": "user", "content": "What is the weather like in Boston?"}
+]
+
+def get_current_weather(location):
+  if location == "Boston, MA":
+    return "The weather is 12F"
+
+functions = [
+    {
+      "name": "get_current_weather",
+      "description": "Get the current weather in a given location",
+      "parameters": {
+        "type": "object",
+        "properties": {
+          "location": {
+            "type": "string",
+            "description": "The city and state, e.g. San Francisco, CA"
+          },
+          "unit": {
+            "type": "string",
+            "enum": ["celsius", "fahrenheit"]
+          }
+        },
+        "required": ["location"]
+      }
+    }
+  ]
+
+response = completion(model="claude-2", messages=messages, functions=functions)
+print(response)
+```
+

docs/my-website/docs/completion/input.md

@@ -0,0 +1,582 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Input Params
+
+## Common Params 
+LiteLLM accepts and translates the [OpenAI Chat Completion params](https://platform.openai.com/docs/api-reference/chat/create) across all providers. 
+
+### Usage
+```python
+import litellm
+
+# set env variables
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+
+## SET MAX TOKENS - via completion() 
+response = litellm.completion(
+            model="gpt-3.5-turbo",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+print(response)
+```
+
+### Translated OpenAI params
+This is a list of openai params we translate across providers.
+
+This list is constantly being updated.
+
+| Provider | temperature | max_tokens | top_p | stream | stop | n | presence_penalty | frequency_penalty | functions | function_call |
+|---|---|---|---|---|---|---|---|---|---|---|
+|Anthropic| ✅ | ✅ | ✅ | ✅ | ✅ |  |  |   |  |   |
+|OpenAI| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+|Replicate | ✅ | ✅ | ✅ | ✅ | ✅ | |  |   |  |   |
+|Anyscale | ✅ | ✅ | ✅ | ✅ |
+|Cohere| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |   |   |
+|Huggingface| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |   |    |
+|Openrouter| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+|AI21| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |   |
+|VertexAI| ✅ | ✅ |  | ✅ |  |  |  |  |  |   |
+|Bedrock| ✅ | ✅ | ✅ | ✅ | ✅ |  |  |   |  |   |
+|Sagemaker| ✅ | ✅ (only `jumpstart llama2`) |  | ✅ |  |  |  |  |  |   |
+|TogetherAI| ✅ | ✅ | ✅ | ✅ | ✅ |  |  |   |  |   |
+|AlephAlpha| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |   |  |   |
+|Palm| ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |  |  |  |   |
+|NLP Cloud| ✅ | ✅ | ✅ | ✅ | ✅ | |  |  |  |   |
+|Petals| ✅ | ✅ |  | ✅ | |  |   |  |  |   |
+|Ollama| ✅ | ✅ | ✅ | ✅ | ✅ |  |   | ✅ |  |   |
+
+:::note
+
+By default, LiteLLM raises an exception if the openai param being passed in isn't supported. 
+
+To drop the param instead, set `litellm.drop_params = True`.
+
+**For function calling:**
+
+Add to prompt for non-openai models, set: `litellm.add_function_to_prompt = True`. 
+::: 
+
+## Input Params
+
+```python
+def completion(
+    model: str,
+    messages: List = [],
+    # Optional OpenAI params
+    temperature: Optional[float] = None,
+    top_p: Optional[float] = None,
+    n: Optional[int] = None,
+    stream: Optional[bool] = None,
+    stop=None,
+    max_tokens: Optional[float] = None,
+    presence_penalty: Optional[float] = None,
+    frequency_penalty: Optional[float]=None,
+    logit_bias: dict = {},
+    user: str = "",
+    deployment_id = None,
+    request_timeout: Optional[int] = None,
+    response_format: Optional[dict] = None,
+    seed: Optional[int] = None,
+    tools: Optional[List] = None,
+    tool_choice: Optional[str] = None,
+    functions: List = [],       # soon to be deprecated
+    function_call: str = "",    # soon to be deprecated
+
+    # Optional LiteLLM params
+    api_base: Optional[str] = None,
+    api_version: Optional[str] = None,
+    api_key: Optional[str] = None,
+    num_retries: Optional[int] = None, # set to retry a model if an APIError, TimeoutError, or ServiceUnavailableError occurs 
+    context_window_fallback_dict: Optional[dict] = None, # mapping of model to use if call fails due to context window error
+    fallbacks: Optional[list] = None, # pass in a list of api_base,keys, etc. 
+    metadata: Optional[dict] = None # additional call metadata, passed to logging integrations / custom callbacks
+    
+
+    **kwargs,
+) -> ModelResponse:
+```
+### Required Fields
+
+- `model`: *string* - ID of the model to use. Refer to the model endpoint compatibility table for details on which models work with the Chat API.
+  
+- `messages`: *array* - A list of messages comprising the conversation so far.
+
+#### Properties of `messages`
+*Note* - Each message in the array contains the following properties:
+
+- `role`: *string* - The role of the message's author. Roles can be: system, user, assistant, or function.
+
+- `content`: *string or null* - The contents of the message. It is required for all messages, but may be null for assistant messages with function calls.
+
+- `name`: *string (optional)* - The name of the author of the message. It is required if the role is "function". The name should match the name of the function represented in the content. It can contain characters (a-z, A-Z, 0-9), and underscores, with a maximum length of 64 characters.
+
+- `function_call`: *object (optional)* - The name and arguments of a function that should be called, as generated by the model.
+
+
+
+## Optional Fields
+
+`temperature`: *number or null (optional)* - The sampling temperature to be used, between 0 and 2. Higher values like 0.8 produce more random outputs, while lower values like 0.2 make outputs more focused and deterministic. 
+
+- `top_p`: *number or null (optional)* - An alternative to sampling with temperature. It instructs the model to consider the results of the tokens with top_p probability. For example, 0.1 means only the tokens comprising the top 10% probability mass are considered.
+
+- `n`: *integer or null (optional)* - The number of chat completion choices to generate for each input message.
+
+- `stream`: *boolean or null (optional)* - If set to true, it sends partial message deltas. Tokens will be sent as they become available, with the stream terminated by a [DONE] message.
+
+- `stop`: *string/ array/ null (optional)* - Up to 4 sequences where the API will stop generating further tokens.
+
+- `max_tokens`: *integer (optional)* - The maximum number of tokens to generate in the chat completion.
+
+- `presence_penalty`: *number or null (optional)* - It is used to penalize new tokens based on their existence in the text so far.
+
+- `response_format`: *object (optional)* - An object specifying the format that the model must output.
+
+    - Setting to `{ "type": "json_object" }` enables JSON mode, which guarantees the message the model generates is valid JSON.
+    
+    - Important: when using JSON mode, you must also instruct the model to produce JSON yourself via a system or user message. Without this, the model may generate an unending stream of whitespace until the generation reaches the token limit, resulting in a long-running and seemingly "stuck" request. Also note that the message content may be partially cut off if finish_reason="length", which indicates the generation exceeded max_tokens or the conversation exceeded the max context length.
+
+- `seed`: *integer or null (optional)* - This feature is in Beta. If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the `system_fingerprint` response parameter to monitor changes in the backend.
+
+- `tools`: *array (optional)* - A list of tools the model may call. Currently, only functions are supported as a tool. Use this to provide a list of functions the model may generate JSON inputs for.
+
+    - `type`: *string* - The type of the tool. Currently, only function is supported.
+
+    - `function`: *object* - Required.
+
+- `tool_choice`: *string or object (optional)* - Controls which (if any) function is called by the model. none means the model will not call a function and instead generates a message. auto means the model can pick between generating a message or calling a function. Specifying a particular function via {"type: "function", "function": {"name": "my_function"}} forces the model to call that function.
+
+    - `none` is the default when no functions are present. `auto` is the default if functions are present.
+
+- `frequency_penalty`: *number or null (optional)* - It is used to penalize new tokens based on their frequency in the text so far.
+
+- `logit_bias`: *map (optional)* - Used to modify the probability of specific tokens appearing in the completion.
+
+- `user`: *string (optional)* - A unique identifier representing your end-user. This can help OpenAI to monitor and detect abuse.
+
+- `timeout`: *int (optional)* - Timeout in seconds for completion requests (Defaults to 600 seconds)
+
+#### Deprecated Params
+- `functions`: *array* - A list of functions that the model may use to generate JSON inputs. Each function should have the following properties:
+
+    - `name`: *string* - The name of the function to be called. It should contain a-z, A-Z, 0-9, underscores and dashes, with a maximum length of 64 characters.
+    
+    - `description`: *string (optional)* - A description explaining what the function does. It helps the model to decide when and how to call the function.
+    
+    - `parameters`: *object* - The parameters that the function accepts, described as a JSON Schema object.
+    
+- `function_call`: *string or object (optional)* - Controls how the model responds to function calls.
+
+
+#### litellm-specific params 
+
+- `api_base`: *string (optional)* - The api endpoint you want to call the model with
+
+- `api_version`: *string (optional)* - (Azure-specific) the api version for the call
+
+- `num_retries`: *int (optional)* - The number of times to retry the API call if an APIError, TimeoutError or ServiceUnavailableError occurs 
+
+- `context_window_fallback_dict`: *dict (optional)* - A mapping of model to use if call fails due to context window error
+
+- `fallbacks`: *list (optional)* - A list of model names + params to be used, in case the initial call fails
+
+- `metadata`: *dict (optional)* - Any additional data you want to be logged when the call is made (sent to logging integrations, eg. promptlayer and accessible via custom callback function)
+
+## Provider-specific Params
+Providers might offer params not supported by OpenAI (e.g. top_k). You can pass those in 2 ways: 
+- via completion(): We'll pass the non-openai param, straight to the provider as part of the request body.
+    - e.g. `completion(model="claude-instant-1", top_k=3)`
+- via provider-specific config variable (e.g. `litellm.OpenAIConfig()`). 
+
+<Tabs>
+<TabItem value="openai" label="OpenAI">
+
+```python
+import litellm, os
+
+# set env variables
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+
+## SET MAX TOKENS - via completion() 
+response_1 = litellm.completion(
+            model="gpt-3.5-turbo",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.OpenAIConfig(max_tokens=10)
+
+response_2 = litellm.completion(
+            model="gpt-3.5-turbo",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+<TabItem value="openai-text" label="OpenAI Text Completion">
+
+```python
+import litellm, os
+
+# set env variables
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+
+
+## SET MAX TOKENS - via completion() 
+response_1 = litellm.completion(
+            model="text-davinci-003",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.OpenAITextCompletionConfig(max_tokens=10)
+response_2 = litellm.completion(
+            model="text-davinci-003",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+<TabItem value="azure-openai" label="Azure OpenAI">
+
+```python
+import litellm, os
+
+# set env variables
+os.environ["AZURE_API_BASE"] = "your-azure-api-base"
+os.environ["AZURE_API_TYPE"] = "azure" # [OPTIONAL] 
+os.environ["AZURE_API_VERSION"] = "2023-07-01-preview" # [OPTIONAL]
+
+## SET MAX TOKENS - via completion() 
+response_1 = litellm.completion(
+            model="azure/chatgpt-v-2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.AzureOpenAIConfig(max_tokens=10)
+response_2 = litellm.completion(
+            model="azure/chatgpt-v-2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+<TabItem value="anthropic" label="Anthropic">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="claude-instant-1",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.AnthropicConfig(max_tokens_to_sample=200)
+response_2 = litellm.completion(
+            model="claude-instant-1",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="huggingface" label="Huggingface">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["HUGGINGFACE_API_KEY"] = "your-huggingface-key" #[OPTIONAL]
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="huggingface/mistralai/Mistral-7B-Instruct-v0.1",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            api_base="https://your-huggingface-api-endpoint",
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.HuggingfaceConfig(max_new_tokens=200)
+response_2 = litellm.completion(
+            model="huggingface/mistralai/Mistral-7B-Instruct-v0.1",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            api_base="https://your-huggingface-api-endpoint"
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="together_ai" label="TogetherAI">
+
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["TOGETHERAI_API_KEY"] = "your-togetherai-key" 
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="together_ai/togethercomputer/llama-2-70b-chat",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.TogetherAIConfig(max_tokens_to_sample=200)
+response_2 = litellm.completion(
+            model="together_ai/togethercomputer/llama-2-70b-chat",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="ollama" label="Ollama">
+
+```python
+import litellm, os 
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="ollama/llama2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.OllamConfig(num_predict=200)
+response_2 = litellm.completion(
+            model="ollama/llama2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="replicate" label="Replicate">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["REPLICATE_API_KEY"] = "your-replicate-key" 
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="replicate/meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.ReplicateConfig(max_new_tokens=200)
+response_2 = litellm.completion(
+            model="replicate/meta/llama-2-70b-chat:02e509c789964a7ea8736978a43525956ef40397be9033abf9fd2badfe68c9e3",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="petals" label="Petals">
+
+
+```python
+import litellm
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="petals/petals-team/StableBeluga2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            api_base="https://chat.petals.dev/api/v1/generate",
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.PetalsConfig(max_new_tokens=10)
+response_2 = litellm.completion(
+            model="petals/petals-team/StableBeluga2",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            api_base="https://chat.petals.dev/api/v1/generate",
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="palm" label="Palm">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["PALM_API_KEY"] = "your-palm-key"  
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="palm/chat-bison",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.PalmConfig(maxOutputTokens=10)
+response_2 = litellm.completion(
+            model="palm/chat-bison",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+</TabItem>
+
+<TabItem value="ai21" label="AI21">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["AI21_API_KEY"] = "your-ai21-key"  
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="j2-mid",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.AI21Config(maxOutputTokens=10)
+response_2 = litellm.completion(
+            model="j2-mid",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+<TabItem value="cohere" label="Cohere">
+
+```python
+import litellm, os 
+
+# set env variables
+os.environ["COHERE_API_KEY"] = "your-cohere-key"   
+
+## SET MAX TOKENS - via completion()
+response_1 = litellm.completion(
+            model="command-nightly",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+            max_tokens=10
+        )
+
+response_1_text = response_1.choices[0].message.content
+
+## SET MAX TOKENS - via config
+litellm.CohereConfig(max_tokens=200)
+response_2 = litellm.completion(
+            model="command-nightly",
+            messages=[{ "content": "Hello, how are you?","role": "user"}],
+        )
+
+response_2_text = response_2.choices[0].message.content
+
+## TEST OUTPUT
+assert len(response_2_text) > len(response_1_text)
+```
+
+</TabItem>
+
+</Tabs>
+
+
+[**Check out the tutorial!**](../tutorials/provider_specific_params.md)

docs/my-website/docs/completion/message_trimming.md

@@ -0,0 +1,36 @@
+# Trimming Input Messages
+**Use litellm.trim_messages() to ensure messages does not exceed a model's token limit or specified `max_tokens`**
+
+## Usage 
+```python
+from litellm import completion
+from litellm.utils import trim_messages
+
+response = completion(
+    model=model, 
+    messages=trim_messages(messages, model) # trim_messages ensures tokens(messages) < max_tokens(model)
+) 
+```
+
+## Usage - set max_tokens
+```python
+from litellm import completion
+from litellm.utils import trim_messages
+
+response = completion(
+    model=model, 
+    messages=trim_messages(messages, model, max_tokens=10), # trim_messages ensures tokens(messages) < max_tokens
+) 
+```
+
+## Parameters
+
+The function uses the following parameters:
+
+- `messages`:[Required] This should be a list of input messages 
+
+- `model`:[Optional] This is the LiteLLM model being used. This parameter is optional, as you can alternatively specify the `max_tokens` parameter.
+
+- `max_tokens`:[Optional] This is an int, manually set upper limit on messages
+
+- `trim_ratio`:[Optional] This represents the target ratio of tokens to use following trimming. It's default value is 0.75, which implies that messages will be trimmed to utilise about 75%

docs/my-website/docs/completion/mock_requests.md

@@ -0,0 +1,72 @@
+# Mock Completion() Responses - Save Testing Costs 💰
+
+For testing purposes, you can use `completion()` with `mock_response` to mock calling the completion endpoint. 
+
+This will return a response object with a default response (works for streaming as well), without calling the LLM APIs. 
+
+## quick start
+```python
+from litellm import completion 
+
+model = "gpt-3.5-turbo"
+messages = [{"role":"user", "content":"This is a test request"}]
+
+completion(model=model, messages=messages, mock_response="It's simple to use and easy to get started")
+```
+
+## streaming
+
+```python
+from litellm import completion 
+model = "gpt-3.5-turbo"
+messages = [{"role": "user", "content": "Hey, I'm a mock request"}]
+response = completion(model=model, messages=messages, stream=True, mock_response="It's simple to use and easy to get started")
+for chunk in response: 
+    print(chunk) # {'choices': [{'delta': {'role': 'assistant', 'content': 'Thi'}, 'finish_reason': None}]}
+    complete_response += chunk["choices"][0]["delta"]["content"]
+```
+
+## (Non-streaming) Mock Response Object 
+
+```json
+{
+  "choices": [
+    {
+      "finish_reason": "stop",
+      "index": 0,
+      "message": {
+        "content": "This is a mock request",
+        "role": "assistant",
+        "logprobs": null
+      }
+    }
+  ],
+  "created": 1694459929.4496052,
+  "model": "MockResponse",
+  "usage": {
+    "prompt_tokens": null,
+    "completion_tokens": null,
+    "total_tokens": null
+  }
+}
+```
+
+## Building a pytest function using `completion` with `mock_response`
+
+```python
+from litellm import completion
+import pytest
+
+def test_completion_openai():
+    try:
+        response = completion(
+            model="gpt-3.5-turbo",
+            messages=[{"role":"user", "content":"Why is LiteLLM amazing?"}],
+            mock_response="LiteLLM is awesome"
+        )
+        # Add any assertions here to check the response
+        print(response)
+        assert(response['choices'][0]['message']['content'] == "LiteLLM is awesome")
+    except Exception as e:
+        pytest.fail(f"Error occurred: {e}")
+```

docs/my-website/docs/completion/model_alias.md

@@ -0,0 +1,53 @@
+# Model Alias
+
+The model name you show an end-user might be different from the one you pass to LiteLLM - e.g. Displaying `GPT-3.5` while calling `gpt-3.5-turbo-16k` on the backend. 
+
+LiteLLM simplifies this by letting you pass in a model alias mapping. 
+
+# expected format
+
+```python
+litellm.model_alias_map = {
+    # a dictionary containing a mapping of the alias string to the actual litellm model name string
+    "model_alias": "litellm_model_name"
+}
+```
+
+# usage 
+
+### Relevant Code
+```python
+model_alias_map = {
+    "GPT-3.5": "gpt-3.5-turbo-16k",
+    "llama2": "replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf"
+}
+
+litellm.model_alias_map = model_alias_map
+```
+
+### Complete Code
+```python
+import litellm 
+from litellm import completion 
+
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "openai key"
+os.environ["REPLICATE_API_KEY"] = "cohere key"
+
+## set model alias map
+model_alias_map = {
+    "GPT-3.5": "gpt-3.5-turbo-16k",
+    "llama2": "replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf"
+}
+
+litellm.model_alias_map = model_alias_map
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# call "gpt-3.5-turbo-16k"
+response = completion(model="GPT-3.5", messages=messages)
+
+# call replicate/llama-2-70b-chat:2796ee9483c3fd7aa2e171d38f4ca1...
+response = completion("llama2", messages)
+```

docs/my-website/docs/completion/multiple_deployments.md

@@ -0,0 +1,53 @@
+# Multiple Deployments
+
+If you have multiple deployments of the same model, you can pass the list of deployments, and LiteLLM will return the first result. 
+
+## Quick Start
+
+Multiple providers offer Mistral-7B-Instruct. 
+
+Here's how you can use litellm to return the first result: 
+
+```python
+from litellm import completion
+
+messages=[{"role": "user", "content": "Hey, how's it going?"}]
+
+## All your mistral deployments ##
+model_list = [{
+	"model_name": "mistral-7b-instruct",
+	"litellm_params": { # params for litellm completion/embedding call 
+        "model": "replicate/mistralai/mistral-7b-instruct-v0.1:83b6a56e7c828e667f21fd596c338fd4f0039b46bcfa18d973e8e70e455fda70", 
+        "api_key": "replicate_api_key",
+    }
+}, {
+	"model_name": "mistral-7b-instruct",
+	"litellm_params": { # params for litellm completion/embedding call 
+        "model": "together_ai/mistralai/Mistral-7B-Instruct-v0.1", 
+        "api_key": "togetherai_api_key",
+    }
+}, {
+	"model_name": "mistral-7b-instruct",
+	"litellm_params": { # params for litellm completion/embedding call 
+        "model": "together_ai/mistralai/Mistral-7B-Instruct-v0.1", 
+        "api_key": "togetherai_api_key",
+    }
+}, {
+	"model_name": "mistral-7b-instruct",
+	"litellm_params": { # params for litellm completion/embedding call 
+        "model": "perplexity/mistral-7b-instruct", 
+		"api_key": "perplexity_api_key"
+    }
+}, {
+	"model_name": "mistral-7b-instruct",
+	"litellm_params": {
+		"model": "deepinfra/mistralai/Mistral-7B-Instruct-v0.1",
+		"api_key": "deepinfra_api_key"
+	}
+}]
+
+## LiteLLM completion call ## returns first response 
+response = completion(model="mistral-7b-instruct", messages=messages, model_list=model_list)
+
+print(response)
+```

docs/my-website/docs/completion/output.md

@@ -0,0 +1,68 @@
+# Output 
+
+## Format
+Here's the exact json output and type you can expect from all litellm `completion` calls for all models
+
+```python 
+{
+  'choices': [
+    {
+      'finish_reason': str,     # String: 'stop'
+      'index': int,             # Integer: 0
+      'message': {              # Dictionary [str, str]
+        'role': str,            # String: 'assistant'
+        'content': str          # String: "default message"
+      }
+    }
+  ],
+  'created': str,               # String: None
+  'model': str,                 # String: None
+  'usage': {                    # Dictionary [str, int]
+    'prompt_tokens': int,       # Integer
+    'completion_tokens': int,   # Integer
+    'total_tokens': int         # Integer
+  }
+}
+
+```
+
+You can access the response as a dictionary or as a class object, just as OpenAI allows you
+```python
+print(response.choices[0].message.content)
+print(response['choices'][0]['message']['content'])
+```
+
+Here's what an example response looks like 
+```python
+{
+  'choices': [
+     {
+        'finish_reason': 'stop',
+        'index': 0,
+        'message': {
+           'role': 'assistant',
+            'content': " I'm doing well, thank you for asking. I am Claude, an AI assistant created by Anthropic."
+        }
+      }
+    ],
+ 'created': 1691429984.3852863,
+ 'model': 'claude-instant-1',
+ 'usage': {'prompt_tokens': 18, 'completion_tokens': 23, 'total_tokens': 41}
+}
+```
+
+## Additional Attributes
+
+You can also access information like latency. 
+
+```python
+from litellm import completion
+import os
+os.environ["ANTHROPIC_API_KEY"] = "your-api-key"
+
+messages=[{"role": "user", "content": "Hey!"}]
+
+response = completion(model="claude-2", messages=messages)
+
+print(response.response_ms) # 616.25# 616.25
+```

docs/my-website/docs/completion/prompt_formatting.md

@@ -0,0 +1,86 @@
+# Prompt Formatting
+
+LiteLLM automatically translates the OpenAI ChatCompletions prompt format, to other models. You can control this by setting a custom prompt template for a model as well. 
+
+## Huggingface Models 
+
+LiteLLM supports [Huggingface Chat Templates](https://huggingface.co/docs/transformers/main/chat_templating), and will automatically check if your huggingface model has a registered chat template (e.g. [Mistral-7b](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/blob/main/tokenizer_config.json#L32)).
+
+For popular models (e.g. meta-llama/llama2), we have their templates saved as part of the package. 
+
+**Stored Templates**
+
+| Model Name | Works for Models | Completion Call
+| -------- | -------- | -------- |
+| mistralai/Mistral-7B-Instruct-v0.1 | mistralai/Mistral-7B-Instruct-v0.1| `completion(model='huggingface/mistralai/Mistral-7B-Instruct-v0.1', messages=messages, api_base="your_api_endpoint")` |
+| meta-llama/Llama-2-7b-chat | All meta-llama llama2 chat models| `completion(model='huggingface/meta-llama/Llama-2-7b', messages=messages, api_base="your_api_endpoint")` |
+| tiiuae/falcon-7b-instruct | All falcon instruct models | `completion(model='huggingface/tiiuae/falcon-7b-instruct', messages=messages, api_base="your_api_endpoint")` |
+| mosaicml/mpt-7b-chat | All mpt chat models | `completion(model='huggingface/mosaicml/mpt-7b-chat', messages=messages, api_base="your_api_endpoint")` |
+| codellama/CodeLlama-34b-Instruct-hf | All codellama instruct models | `completion(model='huggingface/codellama/CodeLlama-34b-Instruct-hf', messages=messages, api_base="your_api_endpoint")` |
+| WizardLM/WizardCoder-Python-34B-V1.0 | All wizardcoder models | `completion(model='huggingface/WizardLM/WizardCoder-Python-34B-V1.0', messages=messages, api_base="your_api_endpoint")` |
+| Phind/Phind-CodeLlama-34B-v2 | All phind-codellama models | `completion(model='huggingface/Phind/Phind-CodeLlama-34B-v2', messages=messages, api_base="your_api_endpoint")` |
+
+[**Jump to code**](https://github.com/BerriAI/litellm/blob/main/litellm/llms/prompt_templates/factory.py)
+
+## Format Prompt Yourself
+
+You can also format the prompt yourself. Here's how: 
+
+```python 
+import litellm
+# Create your own custom prompt template 
+litellm.register_prompt_template(
+	    model="togethercomputer/LLaMA-2-7B-32K",
+        initial_prompt_value="You are a good assistant" # [OPTIONAL]
+	    roles={
+            "system": {
+                "pre_message": "[INST] <<SYS>>\n", # [OPTIONAL]
+                "post_message": "\n<</SYS>>\n [/INST]\n" # [OPTIONAL]
+            },
+            "user": { 
+                "pre_message": "[INST] ", # [OPTIONAL]
+                "post_message": " [/INST]" # [OPTIONAL]
+            }, 
+            "assistant": {
+                "pre_message": "\n" # [OPTIONAL]
+                "post_message": "\n" # [OPTIONAL]
+            }
+        }
+        final_prompt_value="Now answer as best you can:" # [OPTIONAL]
+)
+
+def test_huggingface_custom_model():
+    model = "huggingface/togethercomputer/LLaMA-2-7B-32K"
+    response = completion(model=model, messages=messages, api_base="https://my-huggingface-endpoint")
+    print(response['choices'][0]['message']['content'])
+    return response
+
+test_huggingface_custom_model()
+```
+
+This is currently supported for Huggingface, TogetherAI, Ollama, and Petals. 
+
+Other providers either have fixed prompt templates (e.g. Anthropic), or format it themselves (e.g. Replicate). If there's a provider we're missing coverage for, let us know! 
+
+## All Providers
+
+Here's the code for how we format all providers. Let us know how we can improve this further
+
+
+| Provider | Model Name | Code |
+| -------- | -------- | -------- |
+| Anthropic | `claude-instant-1`, `claude-instant-1.2`, `claude-2` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/anthropic.py#L84)
+| OpenAI Text Completion | `text-davinci-003`, `text-curie-001`, `text-babbage-001`, `text-ada-001`, `babbage-002`, `davinci-002`, | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/main.py#L442)
+| Replicate | all model names starting with `replicate/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/replicate.py#L180)
+| Cohere | `command-nightly`, `command`, `command-light`, `command-medium-beta`, `command-xlarge-beta` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/cohere.py#L115)
+| Huggingface | all model names starting with `huggingface/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/huggingface_restapi.py#L186)
+| OpenRouter | all model names starting with `openrouter/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/main.py#L611)
+| AI21 | `j2-mid`, `j2-light`, `j2-ultra` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/ai21.py#L107)
+| VertexAI | `text-bison`, `text-bison@001`, `chat-bison`, `chat-bison@001`, `chat-bison-32k`, `code-bison`, `code-bison@001`, `code-gecko@001`, `code-gecko@latest`, `codechat-bison`, `codechat-bison@001`, `codechat-bison-32k` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/vertex_ai.py#L89)
+| Bedrock | all model names starting with `bedrock/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/bedrock.py#L183)
+| Sagemaker | `sagemaker/jumpstart-dft-meta-textgeneration-llama-2-7b` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/sagemaker.py#L89)
+| TogetherAI | all model names starting with `together_ai/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/together_ai.py#L101)
+| AlephAlpha | all model names starting with `aleph_alpha/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/aleph_alpha.py#L184)
+| Palm | all model names starting with `palm/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/palm.py#L95)
+| NLP Cloud | all model names starting with `palm/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/nlp_cloud.py#L120)
+| Petals | all model names starting with `petals/` | [Code](https://github.com/BerriAI/litellm/blob/721564c63999a43f96ee9167d0530759d51f8d45/litellm/llms/petals.py#L87)

docs/my-website/docs/completion/reliable_completions.md

@@ -0,0 +1,196 @@
+# Reliability - Retries, Fallbacks
+
+LiteLLM helps prevent failed requests in 2 ways: 
+- Retries
+- Fallbacks: Context Window + General
+
+## Helper utils 
+LiteLLM supports the following functions for reliability:
+* `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents  
+* `num_retries`: use tenacity retries
+* `completion()` with fallbacks: switch between models/keys/api bases in case of errors. 
+
+## Retry failed requests
+
+Call it in completion like this `completion(..num_retries=2)`.
+
+
+Here's a quick look at how you can use it: 
+
+```python 
+from litellm import completion
+
+user_message = "Hello, whats the weather in San Francisco??"
+messages = [{"content": user_message, "role": "user"}]
+
+# normal call 
+response = completion(
+            model="gpt-3.5-turbo",
+            messages=messages,
+            num_retries=2
+        )
+```
+
+## Fallbacks 
+
+### Context Window Fallbacks
+```python 
+from litellm import completion
+
+fallback_dict = {"gpt-3.5-turbo": "gpt-3.5-turbo-16k"}
+messages = [{"content": "how does a court case get to the Supreme Court?" * 500, "role": "user"}]
+
+completion(model="gpt-3.5-turbo", messages=messages, context_window_fallback_dict=ctx_window_fallback_dict)
+```
+
+### Fallbacks - Switch Models/API Keys/API Bases
+
+LLM APIs can be unstable, completion() with fallbacks ensures you'll always get a response from your calls
+
+#### Usage 
+To use fallback models with `completion()`, specify a list of models in the `fallbacks` parameter. 
+
+The `fallbacks` list should include the primary model you want to use, followed by additional models that can be used as backups in case the primary model fails to provide a response.
+
+#### switch models 
+```python
+response = completion(model="bad-model", messages=messages, 
+    fallbacks=["gpt-3.5-turbo" "command-nightly"])
+```
+
+#### switch api keys/bases (E.g. azure deployment)
+Switch between different keys for the same azure deployment, or use another deployment as well. 
+
+```python
+api_key="bad-key"
+response = completion(model="azure/gpt-4", messages=messages, api_key=api_key,
+    fallbacks=[{"api_key": "good-key-1"}, {"api_key": "good-key-2", "api_base": "good-api-base-2"}])
+```
+
+[Check out this section for implementation details](#fallbacks-1)
+
+## Implementation Details 
+
+### Fallbacks
+#### Output from calls
+```
+Completion with 'bad-model': got exception Unable to map your input to a model. Check your input - {'model': 'bad-model'
+
+
+
+completion call gpt-3.5-turbo
+{
+  "id": "chatcmpl-7qTmVRuO3m3gIBg4aTmAumV1TmQhB",
+  "object": "chat.completion",
+  "created": 1692741891,
+  "model": "gpt-3.5-turbo-0613",
+  "choices": [
+    {
+      "index": 0,
+      "message": {
+        "role": "assistant",
+        "content": "I apologize, but as an AI, I do not have the capability to provide real-time weather updates. However, you can easily check the current weather in San Francisco by using a search engine or checking a weather website or app."
+      },
+      "finish_reason": "stop"
+    }
+  ],
+  "usage": {
+    "prompt_tokens": 16,
+    "completion_tokens": 46,
+    "total_tokens": 62
+  }
+}
+
+```
+
+#### How does fallbacks work
+
+When you pass `fallbacks` to `completion`, it makes the first `completion` call using the primary model specified as `model` in `completion(model=model)`. If the primary model fails or encounters an error, it automatically tries the `fallbacks` models in the specified order. This ensures a response even if the primary model is unavailable.
+
+
+#### Key components of Model Fallbacks implementation:
+* Looping through `fallbacks`
+* Cool-Downs for rate-limited models
+
+#### Looping through `fallbacks`
+Allow `45seconds` for each request. In the 45s this function tries calling the primary model set as `model`. If model fails it loops through the backup `fallbacks` models and attempts to get a response in the allocated `45s` time set here: 
+```python
+while response == None and time.time() - start_time < 45:
+        for model in fallbacks:
+```
+
+#### Cool-Downs for rate-limited models
+If a model API call leads to an error - allow it to cooldown for `60s`
+```python
+except Exception as e:
+  print(f"got exception {e} for model {model}")
+  rate_limited_models.add(model)
+  model_expiration_times[model] = (
+      time.time() + 60
+  )  # cool down this selected model
+  pass
+```
+
+Before making an LLM API call we check if the selected model is in `rate_limited_models`, if so skip making the API call
+```python
+if (
+  model in rate_limited_models
+):  # check if model is currently cooling down
+  if (
+      model_expiration_times.get(model)
+      and time.time() >= model_expiration_times[model]
+  ):
+      rate_limited_models.remove(
+          model
+      )  # check if it's been 60s of cool down and remove model
+  else:
+      continue  # skip model
+
+```
+
+#### Full code of completion with fallbacks()
+```python
+
+    response = None
+    rate_limited_models = set()
+    model_expiration_times = {}
+    start_time = time.time()
+    fallbacks = [kwargs["model"]] + kwargs["fallbacks"]
+    del kwargs["fallbacks"]  # remove fallbacks so it's not recursive
+
+    while response == None and time.time() - start_time < 45:
+        for model in fallbacks:
+            # loop thru all models
+            try:
+                if (
+                    model in rate_limited_models
+                ):  # check if model is currently cooling down
+                    if (
+                        model_expiration_times.get(model)
+                        and time.time() >= model_expiration_times[model]
+                    ):
+                        rate_limited_models.remove(
+                            model
+                        )  # check if it's been 60s of cool down and remove model
+                    else:
+                        continue  # skip model
+
+                # delete model from kwargs if it exists
+                if kwargs.get("model"):
+                    del kwargs["model"]
+
+                print("making completion call", model)
+                response = litellm.completion(**kwargs, model=model)
+
+                if response != None:
+                    return response
+
+            except Exception as e:
+                print(f"got exception {e} for model {model}")
+                rate_limited_models.add(model)
+                model_expiration_times[model] = (
+                    time.time() + 60
+                )  # cool down this selected model
+                pass
+    return response
+```

docs/my-website/docs/completion/stream.md

@@ -0,0 +1,76 @@
+# Streaming + Async
+
+- [Streaming Responses](#streaming-responses)
+- [Async Completion](#async-completion)
+- [Async + Streaming Completion](#async-streaming)
+
+## Streaming Responses
+LiteLLM supports streaming the model response back by passing `stream=True` as an argument to the completion function
+### Usage
+```python
+from litellm import completion
+messages = [{"role": "user", "content": "Hey, how's it going?"}]
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+for part in response:
+    print(part.choices[0].delta.content or "")
+```
+
+### Helper function
+
+LiteLLM also exposes a helper function to rebuild the complete streaming response from the list of chunks. 
+
+```python
+from litellm import completion
+messages = [{"role": "user", "content": "Hey, how's it going?"}]
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+
+for chunk in response: 
+    chunks.append(chunk)
+
+print(litellm.stream_chunk_builder(chunks, messages=messages))
+```
+
+## Async Completion
+Asynchronous Completion with LiteLLM. LiteLLM provides an asynchronous version of the completion function called `acompletion`
+### Usage
+```python
+from litellm import acompletion
+import asyncio
+
+async def test_get_response():
+    user_message = "Hello, how are you?"
+    messages = [{"content": user_message, "role": "user"}]
+    response = await acompletion(model="gpt-3.5-turbo", messages=messages)
+    return response
+
+response = asyncio.run(test_get_response())
+print(response)
+
+```
+
+## Async Streaming
+We've implemented an `__anext__()` function in the streaming object returned. This enables async iteration over the streaming object. 
+
+### Usage
+Here's an example of using it with openai.
+```python
+from litellm import acompletion
+import asyncio, os, traceback
+
+async def completion_call():
+    try:
+        print("test acompletion + streaming")
+        response = await acompletion(
+            model="gpt-3.5-turbo", 
+            messages=[{"content": "Hello, how are you?", "role": "user"}], 
+            stream=True
+        )
+        print(f"response: {response}")
+        async for chunk in response:
+            print(chunk)
+    except:
+        print(f"error occurred: {traceback.format_exc()}")
+        pass
+
+asyncio.run(completion_call())
+```

docs/my-website/docs/completion/token_usage.md

@@ -0,0 +1,154 @@
+# Completion Token Usage & Cost
+By default LiteLLM returns token usage in all completion requests ([See here](https://litellm.readthedocs.io/en/latest/output/))
+
+However, we also expose 5 helper functions + **[NEW]** an API to calculate token usage across providers:
+
+- `encode`: This encodes the text passed in, using the model-specific tokenizer. [**Jump to code**](#1-encode)
+
+- `decode`: This decodes the tokens passed in, using the model-specific tokenizer. [**Jump to code**](#2-decode)
+
+- `token_counter`: This returns the number of tokens for a given input - it uses the tokenizer based on the model, and defaults to tiktoken if no model-specific tokenizer is available. [**Jump to code**](#3-token_counter)
+
+- `cost_per_token`: This returns the cost (in USD) for prompt (input) and completion (output) tokens. Uses the live list from `api.litellm.ai`. [**Jump to code**](#4-cost_per_token)
+
+- `completion_cost`: This returns the overall cost (in USD) for a given LLM API Call. It combines `token_counter` and `cost_per_token` to return the cost for that query (counting both cost of input and output). [**Jump to code**](#5-completion_cost)
+
+- `get_max_tokens`: This returns the maximum number of tokens allowed for the given model. [**Jump to code**](#6-get_max_tokens)
+
+- `model_cost`: This returns a dictionary for all models, with their max_tokens, input_cost_per_token and output_cost_per_token. It uses the `api.litellm.ai` call shown below. [**Jump to code**](#7-model_cost)
+
+- `register_model`: This registers new / overrides existing models (and their pricing details) in the model cost dictionary. [**Jump to code**](#8-register_model)
+
+- `api.litellm.ai`: Live token + price count across [all supported models](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json). [**Jump to code**](#9-apilitellmai)
+
+📣 This is a community maintained list. Contributions are welcome! ❤️
+
+## Example Usage 
+
+### 1. `encode`
+Encoding has model-specific tokenizers for anthropic, cohere, llama2 and openai. If an unsupported model is passed in, it'll default to using tiktoken (openai's tokenizer).
+
+```python
+from litellm import encode, decode
+
+sample_text = "Hellö World, this is my input string!"
+# openai encoding + decoding
+openai_tokens = encode(model="gpt-3.5-turbo", text=sample_text)
+print(openai_tokens)
+```
+
+### 2. `decode`
+
+Decoding is supported for anthropic, cohere, llama2 and openai.
+
+```python
+from litellm import encode, decode
+
+sample_text = "Hellö World, this is my input string!"
+# openai encoding + decoding
+openai_tokens = encode(model="gpt-3.5-turbo", text=sample_text)
+openai_text = decode(model="gpt-3.5-turbo", tokens=openai_tokens)
+print(openai_text)
+```
+
+### 3. `token_counter`
+
+```python
+from litellm import token_counter
+
+messages = [{"user": "role", "content": "Hey, how's it going"}]
+print(token_counter(model="gpt-3.5-turbo", messages=messages))
+```
+
+### 4. `cost_per_token`
+
+```python
+from litellm import cost_per_token
+
+prompt_tokens =  5
+completion_tokens = 10
+prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar = cost_per_token(model="gpt-3.5-turbo", prompt_tokens=prompt_tokens, completion_tokens=completion_tokens))
+
+print(prompt_tokens_cost_usd_dollar, completion_tokens_cost_usd_dollar)
+```
+
+### 5. `completion_cost`
+
+* Input: Accepts a `litellm.completion()` response **OR** prompt + completion strings
+* Output: Returns a `float` of cost for the `completion` call 
+
+**litellm.completion()**
+```python
+from litellm import completion, completion_cost
+
+response = completion(
+            model="bedrock/anthropic.claude-v2",
+            messages=messages,
+            request_timeout=200,
+        )
+# pass your response from completion to completion_cost
+cost = completion_cost(completion_response=response)
+formatted_string = f"${float(cost):.10f}"
+print(formatted_string)
+```
+
+**prompt + completion string**
+```python
+from litellm import completion_cost
+cost = completion_cost(model="bedrock/anthropic.claude-v2", prompt="Hey!", completion="How's it going?")
+formatted_string = f"${float(cost):.10f}"
+print(formatted_string)
+```
+### 6. `get_max_tokens`
+
+Input: Accepts a model name - e.g., gpt-3.5-turbo (to get a complete list, call litellm.model_list).
+Output: Returns the maximum number of tokens allowed for the given model
+
+```python 
+from litellm import get_max_tokens 
+
+model = "gpt-3.5-turbo"
+
+print(get_max_tokens(model)) # Output: 4097
+```
+
+### 7. `model_cost`
+
+* Output: Returns a dict object containing the max_tokens, input_cost_per_token, output_cost_per_token for all models on [community-maintained list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json)
+
+```python 
+from litellm import model_cost 
+
+print(model_cost) # {'gpt-3.5-turbo': {'max_tokens': 4000, 'input_cost_per_token': 1.5e-06, 'output_cost_per_token': 2e-06}, ...}
+```
+
+### 8. `register_model`
+
+* Input: Provide EITHER a model cost dictionary or a url to a hosted json blob
+* Output: Returns updated model_cost dictionary + updates litellm.model_cost with model details.  
+
+**Dictionary**
+```python
+from litellm import register_model
+
+litellm.register_model({
+        "gpt-4": {
+        "max_tokens": 8192, 
+        "input_cost_per_token": 0.00002, 
+        "output_cost_per_token": 0.00006, 
+        "litellm_provider": "openai", 
+        "mode": "chat"
+    },
+})
+```
+
+**URL for json blob**
+```python
+import litellm
+
+litellm.register_model(model_cost=
+"https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json")
+```
+
+
+

docs/my-website/docs/contact.md

@@ -0,0 +1,6 @@
+# Contact Us
+
+[![](https://dcbadge.vercel.app/api/server/wuPM9dRgDw)](https://discord.gg/wuPM9dRgDw)
+
+* [Meet with us 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+* Contact us at [email protected] / [email protected]

docs/my-website/docs/debugging/hosted_debugging.md

@@ -0,0 +1,91 @@
+import Image from '@theme/IdealImage';
+import QueryParamReader from '../../src/components/queryParamReader.js'
+
+# [Beta] Monitor Logs in Production
+
+:::note
+
+This is in beta. Expect frequent updates, as we improve based on your feedback.
+
+:::
+
+LiteLLM provides an integration to let you monitor logs in production.
+
+👉 Jump to our sample LiteLLM Dashboard: https://admin.litellm.ai/
+
+
+<Image img={require('../../img/alt_dashboard.png')} alt="Dashboard" />
+
+## Debug your first logs
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_OpenAI.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+
+### 1. Get your LiteLLM Token
+
+Go to [admin.litellm.ai](https://admin.litellm.ai/) and copy the code snippet with your unique token
+
+<Image img={require('../../img/hosted_debugger_usage_page.png')} alt="Usage" />
+
+### 2. Set up your environment
+
+**Add it to your .env**
+
+```python
+import os 
+
+os.env["LITELLM_TOKEN"] = "e24c4c06-d027-4c30-9e78-18bc3a50aebb" # replace with your unique token
+
+```
+
+**Turn on LiteLLM Client**
+```python
+import litellm 
+litellm.client = True
+```
+
+### 3. Make a normal `completion()` call
+```python
+import litellm 
+from litellm import completion
+import os 
+
+# set env variables
+os.environ["LITELLM_TOKEN"] = "e24c4c06-d027-4c30-9e78-18bc3a50aebb" # replace with your unique token
+os.environ["OPENAI_API_KEY"] = "openai key"
+
+litellm.use_client = True # enable logging dashboard 
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+```
+
+Your `completion()` call print with a link to your session dashboard (https://admin.litellm.ai/<your_unique_token>)
+
+In the above case it would be: [`admin.litellm.ai/e24c4c06-d027-4c30-9e78-18bc3a50aebb`](https://admin.litellm.ai/e24c4c06-d027-4c30-9e78-18bc3a50aebb)
+
+Click on your personal dashboard link. Here's how you can find it 👇
+
+<Image img={require('../../img/dash_output.png')} alt="Dashboard" />
+
+[👋 Tell us if you need better privacy controls](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version?month=2023-08)  
+
+### 3. Review request log 
+
+Oh! Looks like our request was made successfully. Let's click on it and see exactly what got sent to the LLM provider. 
+
+<Image img={require('../../img/dashboard_log_row.png')} alt="Dashboard Log Row" />    
+
+
+
+Ah! So we can see that this request was made to a **Baseten** (see litellm_params > custom_llm_provider) for a model with ID - **7qQNLDB** (see model). The message sent was - `"Hey, how's it going?"` and the response received was - `"As an AI language model, I don't have feelings or emotions, but I can assist you with your queries. How can I assist you today?"`
+
+<Image img={require('../../img/dashboard_log.png')} alt="Dashboard Log Row" />
+
+:::info
+
+🎉 Congratulations! You've successfully debugger your first log!
+
+:::

docs/my-website/docs/debugging/local_debugging.md

@@ -0,0 +1,64 @@
+# Local Debugging
+There's 2 ways to do local debugging - `litellm.set_verbose=True` and by passing in a custom function `completion(...logger_fn=<your_local_function>)`. Warning: Make sure to not use `set_verbose` in production. It logs API keys, which might end up in log files.
+
+## Set Verbose 
+
+This is good for getting print statements for everything litellm is doing.
+```python
+import litellm
+from litellm import completion
+
+litellm.set_verbose=True # 👈 this is the 1-line change you need to make
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "openai key"
+os.environ["COHERE_API_KEY"] = "cohere key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+
+# cohere call
+response = completion("command-nightly", messages)
+```
+
+## Logger Function 
+But sometimes all you care about is seeing exactly what's getting sent to your api call and what's being returned - e.g. if the api call is failing, why is that happening? what are the exact params being set? 
+
+In that case, LiteLLM allows you to pass in a custom logging function to see / modify the model call Input/Outputs. 
+
+**Note**: We expect you to accept a dict object. 
+
+Your custom function 
+
+```python
+def my_custom_logging_fn(model_call_dict):
+    print(f"model call details: {model_call_dict}")
+```
+
+### Complete Example
+```python
+from litellm import completion
+
+def my_custom_logging_fn(model_call_dict):
+    print(f"model call details: {model_call_dict}")
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "openai key"
+os.environ["COHERE_API_KEY"] = "cohere key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages, logger_fn=my_custom_logging_fn)
+
+# cohere call
+response = completion("command-nightly", messages, logger_fn=my_custom_logging_fn)
+```
+
+## Still Seeing Issues? 
+
+Text us @ +17708783106 or Join the [Discord](https://discord.com/invite/wuPM9dRgDw). 
+
+We promise to help you in `lite`ning speed ❤️

docs/my-website/docs/default_code_snippet.md

@@ -0,0 +1,22 @@
+---
+displayed_sidebar: tutorialSidebar
+---
+# Get Started
+
+import QueryParamReader from '../src/components/queryParamReader.js'
+import TokenComponent from '../src/components/queryParamToken.js'
+
+:::info
+
+This section assumes you've already added your API keys in <TokenComponent/>
+
+If you want to use the non-hosted version, [go here](https://docs.litellm.ai/docs/#quick-start)
+
+:::
+
+
+```
+pip install litellm
+```
+
+<QueryParamReader/>

docs/my-website/docs/embedding/async_embedding.md

@@ -0,0 +1,15 @@
+# Async Embedding
+
+LiteLLM provides an asynchronous version of the `embedding` function called `aembedding`
+### Usage
+```python
+from litellm import aembedding
+import asyncio
+
+async def test_get_response():
+    response = await aembedding('text-embedding-ada-002', input=["good morning from litellm"])
+    return response
+
+response = asyncio.run(test_get_response())
+print(response)
+```

docs/my-website/docs/embedding/moderation.md

@@ -0,0 +1,10 @@
+# Moderation
+LiteLLM supports the moderation endpoint for OpenAI
+
+## Usage
+```python
+import os
+from litellm import moderation
+os.environ['OPENAI_API_KEY'] = ""
+response = moderation(input="i'm ishaan cto of litellm")   
+```

docs/my-website/docs/embedding/supported_embedding.md

@@ -0,0 +1,201 @@
+# Embedding Models
+
+## Quick Start
+```python
+from litellm import embedding
+import os
+os.environ['OPENAI_API_KEY'] = ""
+response = embedding(model='text-embedding-ada-002', input=["good morning from litellm"])
+```
+
+### Input Params for `litellm.embedding()`
+### Required Fields
+
+- `model`: *string* - ID of the model to use. `model='text-embedding-ada-002'`
+
+- `input`: *array* - Input text to embed, encoded as a string or array of tokens. To embed multiple inputs in a single request, pass an array of strings or array of token arrays. The input must not exceed the max input tokens for the model (8192 tokens for text-embedding-ada-002), cannot be an empty string, and any array must be 2048 dimensions or less. 
+```
+input=["good morning from litellm"]
+```
+
+### Optional LiteLLM Fields
+
+- `user`: *string (optional)* A unique identifier representing your end-user, 
+
+- `timeout`: *integer* - The maximum time, in seconds, to wait for the API to respond. Defaults to 600 seconds (10 minutes).
+
+- `api_base`: *string (optional)* - The api endpoint you want to call the model with
+
+- `api_version`: *string (optional)* - (Azure-specific) the api version for the call
+
+- `api_key`: *string (optional)* - The API key to authenticate and authorize requests. If not provided, the default API key is used.
+
+- `api_type`: *string (optional)* - The type of API to use.
+
+### Output from `litellm.embedding()`
+
+```json
+{
+  "object": "list",
+  "data": [
+    {
+      "object": "embedding",
+      "index": 0,
+      "embedding": [
+        -0.0022326677571982145,
+        0.010749882087111473,
+        ...
+        ...
+        ...
+   
+      ]
+    }
+  ],
+  "model": "text-embedding-ada-002-v2",
+  "usage": {
+    "prompt_tokens": 10,
+    "total_tokens": 10
+  }
+}
+```
+
+## OpenAI Embedding Models
+
+### Usage
+```python
+from litellm import embedding
+import os
+os.environ['OPENAI_API_KEY'] = ""
+response = embedding('text-embedding-ada-002', input=["good morning from litellm"])
+```
+
+| Model Name           | Function Call                               | Required OS Variables                |
+|----------------------|---------------------------------------------|--------------------------------------|
+| text-embedding-ada-002 | `embedding('text-embedding-ada-002', input)` | `os.environ['OPENAI_API_KEY']`       |
+
+## Azure OpenAI Embedding Models
+
+### API keys
+This can be set as env variables or passed as **params to litellm.embedding()**
+```python
+import os
+os.environ['AZURE_API_KEY'] = 
+os.environ['AZURE_API_BASE'] = 
+os.environ['AZURE_API_VERSION'] = 
+```
+
+### Usage
+```python
+from litellm import embedding
+response = embedding(
+    model="azure/<your deployment name>",
+    input=["good morning from litellm"],
+    api_key=api_key,
+    api_base=api_base,
+    api_version=api_version,
+)
+print(response)
+```
+
+| Model Name           | Function Call                               |
+|----------------------|---------------------------------------------|
+| text-embedding-ada-002 | `embedding(model="azure/<your deployment name>", input=input)` |
+
+h/t to [Mikko](https://www.linkedin.com/in/mikkolehtimaki/) for this integration
+
+## OpenAI Compatible Embedding Models
+Use this for calling `/embedding` endpoints on OpenAI Compatible Servers, example https://github.com/xorbitsai/inference
+
+**Note add `openai/` prefix to model so litellm knows to route to OpenAI**
+
+### Usage
+```python
+from litellm import embedding
+response = embedding(
+  model = "openai/<your-llm-name>",     # add `openai/` prefix to model so litellm knows to route to OpenAI
+  api_base="http://0.0.0.0:8000/"       # set API Base of your Custom OpenAI Endpoint
+  input=["good morning from litellm"]
+)
+```
+
+## Bedrock Embedding
+
+### API keys
+This can be set as env variables or passed as **params to litellm.embedding()**
+```python
+import os
+os.environ["AWS_ACCESS_KEY_ID"] = ""  # Access key
+os.environ["AWS_SECRET_ACCESS_KEY"] = "" # Secret access key
+os.environ["AWS_REGION_NAME"] = "" # us-east-1, us-east-2, us-west-1, us-west-2
+```
+
+### Usage
+```python
+from litellm import embedding
+response = embedding(
+    model="amazon.titan-embed-text-v1",
+    input=["good morning from litellm"],
+)
+print(response)
+```
+
+| Model Name           | Function Call                               |
+|----------------------|---------------------------------------------|
+| Titan Embeddings - G1 | `embedding(model="amazon.titan-embed-text-v1", input=input)` |
+
+
+## Cohere Embedding Models
+https://docs.cohere.com/reference/embed
+
+### Usage
+```python
+from litellm import embedding
+os.environ["COHERE_API_KEY"] = "cohere key"
+
+# cohere call
+response = embedding(
+    model="embed-english-v3.0", 
+    input=["good morning from litellm", "this is another item"], 
+    input_type="search_document" # optional param for v3 llms
+)
+```
+| Model Name               | Function Call                                                |
+|--------------------------|--------------------------------------------------------------|
+| embed-english-v3.0       | `embedding(model="embed-english-v3.0", input=["good morning from litellm", "this is another item"])` |
+| embed-english-light-v3.0 | `embedding(model="embed-english-light-v3.0", input=["good morning from litellm", "this is another item"])` |
+| embed-multilingual-v3.0  | `embedding(model="embed-multilingual-v3.0", input=["good morning from litellm", "this is another item"])` |
+| embed-multilingual-light-v3.0 | `embedding(model="embed-multilingual-light-v3.0", input=["good morning from litellm", "this is another item"])` |
+| embed-english-v2.0       | `embedding(model="embed-english-v2.0", input=["good morning from litellm", "this is another item"])` |
+| embed-english-light-v2.0 | `embedding(model="embed-english-light-v2.0", input=["good morning from litellm", "this is another item"])` |
+| embed-multilingual-v2.0  | `embedding(model="embed-multilingual-v2.0", input=["good morning from litellm", "this is another item"])` |
+
+## HuggingFace Embedding Models
+LiteLLM supports all Feature-Extraction Embedding models: https://huggingface.co/models?pipeline_tag=feature-extraction
+
+### Usage
+```python
+from litellm import embedding
+import os
+os.environ['HUGGINGFACE_API_KEY'] = ""
+response = embedding(
+    model='huggingface/microsoft/codebert-base', 
+    input=["good morning from litellm"]
+)
+```
+### Usage - Custom API Base
+```python
+from litellm import embedding
+import os
+os.environ['HUGGINGFACE_API_KEY'] = ""
+response = embedding(
+    model='huggingface/microsoft/codebert-base', 
+    input=["good morning from litellm"],
+    api_base = "https://p69xlsj6rpno5drq.us-east-1.aws.endpoints.huggingface.cloud"
+)
+```
+
+| Model Name            | Function Call | Required OS Variables                        |
+|-----------------------|--------------------------------------------------------------|-------------------------------------------------|
+| microsoft/codebert-base    | `embedding('huggingface/microsoft/codebert-base', input=input)`               | `os.environ['HUGGINGFACE_API_KEY']`                                             |
+| BAAI/bge-large-zh | `embedding('huggingface/BAAI/bge-large-zh', input=input)`         | `os.environ['HUGGINGFACE_API_KEY']`                                             |
+| any-hf-embedding-model | `embedding('huggingface/hf-embedding-model', input=input)`         | `os.environ['HUGGINGFACE_API_KEY']`                                             |

docs/my-website/docs/exception_mapping.md

@@ -0,0 +1,102 @@
+# Exception Mapping
+
+LiteLLM maps exceptions across all providers to their OpenAI counterparts.
+- Rate Limit Errors
+- Invalid Request Errors
+- Authentication Errors
+- Timeout Errors `openai.APITimeoutError`
+- ServiceUnavailableError 
+- APIError 
+- APIConnectionError
+
+Base case we return APIConnectionError
+
+All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM. 
+
+For all cases, the exception returned inherits from the original OpenAI Exception but contains 3 additional attributes: 
+* status_code - the http status code of the exception
+* message - the error message
+* llm_provider - the provider raising the exception
+
+## Usage
+
+```python 
+import litellm
+import openai
+
+try:
+    response = litellm.completion(
+                model="gpt-4",
+                messages=[
+                    {
+                        "role": "user",
+                        "content": "hello, write a 20 pageg essay"
+                    }
+                ],
+                timeout=0.01, # this will raise a timeout exception
+            )
+except openai.APITimeoutError as e:
+    print("Passed: Raised correct exception. Got openai.APITimeoutError\nGood Job", e)
+    print(type(e))
+    pass
+```
+
+## Usage - Catching Streaming Exceptions
+```python
+import litellm
+try:
+    response = litellm.completion(
+        model="gpt-3.5-turbo",
+        messages=[
+            {
+                "role": "user",
+                "content": "hello, write a 20 pg essay"
+            }
+        ],
+        timeout=0.0001, # this will raise an exception
+        stream=True,
+    )
+    for chunk in response:
+        print(chunk)
+except openai.APITimeoutError as e:
+    print("Passed: Raised correct exception. Got openai.APITimeoutError\nGood Job", e)
+    print(type(e))
+    pass
+except Exception as e:
+    print(f"Did not raise error `openai.APITimeoutError`. Instead raised error type: {type(e)}, Error: {e}")
+
+```
+
+## Details 
+
+To see how it's implemented - [check out the code](https://github.com/BerriAI/litellm/blob/a42c197e5a6de56ea576c73715e6c7c6b19fa249/litellm/utils.py#L1217)
+
+[Create an issue](https://github.com/BerriAI/litellm/issues/new) **or** [make a PR](https://github.com/BerriAI/litellm/pulls) if you want to improve the exception mapping. 
+
+**Note** For OpenAI and Azure we return the original exception (since they're of the OpenAI Error type). But we add the 'llm_provider' attribute to them. [See code](https://github.com/BerriAI/litellm/blob/a42c197e5a6de56ea576c73715e6c7c6b19fa249/litellm/utils.py#L1221)
+
+## Custom mapping list
+
+Base case - we return the original exception.
+
+|               | ContextWindowExceededError | AuthenticationError | InvalidRequestError | RateLimitError | ServiceUnavailableError |
+|---------------|----------------------------|---------------------|---------------------|---------------|-------------------------|
+| Anthropic     | ✅                          | ✅                   | ✅                   | ✅             |                         |
+| OpenAI        | ✅                          | ✅                     |✅                     |✅               |✅|
+| Replicate     | ✅                          | ✅                   | ✅                   | ✅             | ✅                       |
+| Cohere        | ✅                          | ✅                   | ✅                    | ✅             | ✅                        |
+| Huggingface   | ✅                          | ✅                   | ✅                   | ✅             |                         |
+| Openrouter    | ✅                          | ✅                   | ✅                    | ✅             |                         |
+| AI21          | ✅                          | ✅                   | ✅                   | ✅             |                         |
+| VertexAI          |                           |                   |✅                   |             |                         |
+| Bedrock          |                           |                   |✅                   |             |                         |
+| Sagemaker          |                           |                   |✅                   |             |                         |
+| TogetherAI    | ✅                          | ✅                   | ✅                   | ✅             |                         |
+| AlephAlpha    | ✅                          | ✅                   | ✅                   | ✅             | ✅                        |
+
+
+> For a deeper understanding of these exceptions, you can check out [this](https://github.com/BerriAI/litellm/blob/d7e58d13bf9ba9edbab2ab2f096f3de7547f35fa/litellm/utils.py#L1544) implementation for additional insights.
+
+The `ContextWindowExceededError` is a sub-class of `InvalidRequestError`. It was introduced to provide more granularity for exception-handling scenarios. Please refer to [this issue to learn more](https://github.com/BerriAI/litellm/issues/228).
+
+Contributions to improve exception mapping are [welcome](https://github.com/BerriAI/litellm#contributing)

docs/my-website/docs/extras/contributing.md

@@ -0,0 +1,49 @@
+# Contributing to Documentation
+
+This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
+
+Clone litellm 
+```
+git clone https://github.com/BerriAI/litellm.git
+```
+
+### Local setup for locally running docs
+
+#### Installation
+```
+npm install --global yarn
+```
+
+
+### Local Development
+
+```
+cd docs/my-website
+```
+
+Let's Install requirement
+
+```
+yarn
+```
+Run website
+
+```
+yarn start
+```
+Open docs here: [http://localhost:3000/](http://localhost:3000/)
+
+```
+
+This command builds your Markdown files into HTML and starts a development server to browse your documentation. Open up [http://127.0.0.1:8000/](http://127.0.0.1:8000/) in your web browser to see your documentation. You can make changes to your Markdown files and your docs will automatically rebuild.
+
+[Full tutorial here](https://docs.readthedocs.io/en/stable/intro/getting-started-with-mkdocs.html)
+
+### Making changes to Docs
+- All the docs are placed under the `docs` directory
+- If you are adding a new `.md` file or editing the hierarchy edit `mkdocs.yml` in the root of the project
+- After testing your changes, make a change to the `main` branch of [github.com/BerriAI/litellm](https://github.com/BerriAI/litellm)
+
+
+
+

docs/my-website/docs/getting_started.md

@@ -0,0 +1,100 @@
+# Getting Started
+
+import QuickStart from '../src/components/QuickStart.js'
+
+LiteLLM simplifies LLM API calls by mapping them all to the [OpenAI ChatCompletion format](https://platform.openai.com/docs/api-reference/chat). 
+
+## basic usage 
+
+By default we provide a free $10 community-key to try all providers supported on LiteLLM. 
+
+```python
+from litellm import completion
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+os.environ["COHERE_API_KEY"] = "your-api-key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages)
+
+# cohere call
+response = completion("command-nightly", messages)
+```
+
+**Need a dedicated key?**
+Email us @ [email protected]
+
+Next Steps 👉 [Call all supported models - e.g. Claude-2, Llama2-70b, etc.](./proxy_api.md#supported-models)
+
+More details 👉 
+* [Completion() function details](./completion/)
+* [All supported models / providers on LiteLLM](./providers/)
+* [Build your own OpenAI proxy](https://github.com/BerriAI/liteLLM-proxy/tree/main)
+
+## streaming
+
+Same example from before. Just pass in `stream=True` in the completion args. 
+```python
+from litellm import completion
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "openai key"
+os.environ["COHERE_API_KEY"] = "cohere key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+# openai call
+response = completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+
+# cohere call
+response = completion("command-nightly", messages, stream=True)
+
+print(response)
+```
+
+More details 👉 
+* [streaming + async](./completion/stream.md)
+* [tutorial for streaming Llama2 on TogetherAI](./tutorials/TogetherAI_liteLLM.md)
+
+## exception handling 
+
+LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM. 
+
+```python 
+from openai.error import OpenAIError
+from litellm import completion
+
+os.environ["ANTHROPIC_API_KEY"] = "bad-key"
+try: 
+    # some code 
+    completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
+except OpenAIError as e:
+    print(e)
+```
+
+## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
+LiteLLM exposes pre defined callbacks to send data to Langfuse, LLMonitor, Helicone, Promptlayer, Traceloop, Slack
+```python
+from litellm import completion
+
+## set env variables for logging tools
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+os.environ["LLMONITOR_APP_ID"] = "your-llmonitor-app-id"
+
+os.environ["OPENAI_API_KEY"]
+
+# set callbacks
+litellm.success_callback = ["langfuse", "llmonitor"] # log input/output to langfuse, llmonitor, supabase
+
+#openai call
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
+```
+
+More details 👉 
+* [exception mapping](./exception_mapping.md)
+* [retries + model fallbacks for completion()](./completion/reliable_completions.md)
+* [tutorial for model fallbacks with completion()](./tutorials/fallbacks.md)

docs/my-website/docs/index.md

@@ -0,0 +1,402 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# LiteLLM - Getting Started
+
+https://github.com/BerriAI/litellm
+
+import QuickStart from '../src/components/QuickStart.js'
+
+## **Call 100+ LLMs using the same Input/Output Format**
+
+## Basic usage 
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/liteLLM_Getting_Started.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+```shell
+pip install litellm
+```
+<Tabs>
+<TabItem value="openai" label="OpenAI">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}]
+)
+```
+
+</TabItem>
+<TabItem value="anthropic" label="Anthropic">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["ANTHROPIC_API_KEY"] = "your-api-key"
+
+response = completion(
+  model="claude-2", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}]
+)
+```
+
+</TabItem>
+
+<TabItem value="vertex" label="VertexAI">
+
+```python
+from litellm import completion
+import os
+
+# auth: run 'gcloud auth application-default'
+os.environ["VERTEX_PROJECT"] = "hardy-device-386718"
+os.environ["VERTEX_LOCATION"] = "us-central1"
+
+response = completion(
+  model="chat-bison", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}]
+)
+```
+
+</TabItem>
+
+<TabItem value="hugging" label="HuggingFace">
+
+```python
+from litellm import completion 
+import os
+
+os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key" 
+
+# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
+response = completion(
+  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
+  messages=[{ "content": "Hello, how are you?","role": "user"}], 
+  api_base="https://my-endpoint.huggingface.cloud"
+)
+
+print(response)
+```
+
+</TabItem>
+
+<TabItem value="azure" label="Azure OpenAI">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["AZURE_API_KEY"] = ""
+os.environ["AZURE_API_BASE"] = ""
+os.environ["AZURE_API_VERSION"] = ""
+
+# azure call
+response = completion(
+  "azure/<your_deployment_name>", 
+  messages = [{ "content": "Hello, how are you?","role": "user"}]
+)
+```
+
+</TabItem>
+
+
+<TabItem value="ollama" label="Ollama">
+
+```python
+from litellm import completion
+
+response = completion(
+            model="ollama/llama2", 
+            messages = [{ "content": "Hello, how are you?","role": "user"}], 
+            api_base="http://localhost:11434"
+)
+```
+</TabItem>
+<TabItem value="or" label="Openrouter">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENROUTER_API_KEY"] = "openrouter_api_key" 
+
+response = completion(
+  model="openrouter/google/palm-2-chat-bison", 
+  messages = [{ "content": "Hello, how are you?","role": "user"}],
+)
+```
+</TabItem>
+
+</Tabs>
+
+## Streaming
+Set `stream=True` in the `completion` args. 
+<Tabs>
+<TabItem value="openai" label="OpenAI">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}],
+  stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+
+</TabItem>
+<TabItem value="anthropic" label="Anthropic">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["ANTHROPIC_API_KEY"] = "your-api-key"
+
+response = completion(
+  model="claude-2", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}],
+  stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+
+</TabItem>
+
+<TabItem value="vertex" label="VertexAI">
+
+```python
+from litellm import completion
+import os
+
+# auth: run 'gcloud auth application-default'
+os.environ["VERTEX_PROJECT"] = "hardy-device-386718"
+os.environ["VERTEX_LOCATION"] = "us-central1"
+
+response = completion(
+  model="chat-bison", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}],
+  stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+
+</TabItem>
+
+<TabItem value="hugging" label="HuggingFace">
+
+```python
+from litellm import completion 
+import os
+
+os.environ["HUGGINGFACE_API_KEY"] = "huggingface_api_key" 
+
+# e.g. Call 'WizardLM/WizardCoder-Python-34B-V1.0' hosted on HF Inference endpoints
+response = completion(
+  model="huggingface/WizardLM/WizardCoder-Python-34B-V1.0",
+  messages=[{ "content": "Hello, how are you?","role": "user"}], 
+  api_base="https://my-endpoint.huggingface.cloud",
+  stream=True,
+)
+
+
+for chunk in response: 
+  print(chunk)
+```
+
+</TabItem>
+
+<TabItem value="azure" label="Azure OpenAI">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["AZURE_API_KEY"] = ""
+os.environ["AZURE_API_BASE"] = ""
+os.environ["AZURE_API_VERSION"] = ""
+
+# azure call
+response = completion(
+  "azure/<your_deployment_name>", 
+  messages = [{ "content": "Hello, how are you?","role": "user"}],
+  stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+
+</TabItem>
+
+
+<TabItem value="ollama" label="Ollama">
+
+```python
+from litellm import completion
+
+response = completion(
+            model="ollama/llama2", 
+            messages = [{ "content": "Hello, how are you?","role": "user"}], 
+            api_base="http://localhost:11434",
+            stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+</TabItem>
+<TabItem value="or" label="Openrouter">
+
+```python
+from litellm import completion
+import os
+
+## set ENV variables
+os.environ["OPENROUTER_API_KEY"] = "openrouter_api_key" 
+
+response = completion(
+  model="openrouter/google/palm-2-chat-bison", 
+  messages = [{ "content": "Hello, how are you?","role": "user"}],
+  stream=True,
+)
+
+for chunk in response: 
+  print(chunk)
+```
+</TabItem>
+
+</Tabs>
+
+## Exception handling 
+
+LiteLLM maps exceptions across all supported providers to the OpenAI exceptions. All our exceptions inherit from OpenAI's exception types, so any error-handling you have for that, should work out of the box with LiteLLM. 
+
+```python 
+from openai.error import OpenAIError
+from litellm import completion
+
+os.environ["ANTHROPIC_API_KEY"] = "bad-key"
+try: 
+    # some code 
+    completion(model="claude-instant-1", messages=[{"role": "user", "content": "Hey, how's it going?"}])
+except OpenAIError as e:
+    print(e)
+```
+
+## Logging Observability - Log LLM Input/Output ([Docs](https://docs.litellm.ai/docs/observability/callbacks))
+LiteLLM exposes pre defined callbacks to send data to Langfuse, LLMonitor, Helicone, Promptlayer, Traceloop, Slack
+```python
+from litellm import completion
+
+## set env variables for logging tools
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+os.environ["LLMONITOR_APP_ID"] = "your-llmonitor-app-id"
+
+os.environ["OPENAI_API_KEY"]
+
+# set callbacks
+litellm.success_callback = ["langfuse", "llmonitor"] # log input/output to langfuse, llmonitor, supabase
+
+#openai call
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
+```
+
+## Calculate Costs, Usage, Latency
+
+Pass the completion response to `litellm.completion_cost(completion_response=response)` and get the cost
+
+```python
+from litellm import completion, completion_cost
+import os
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{ "content": "Hello, how are you?","role": "user"}]
+)
+
+cost = completion_cost(completion_response=response)
+print("Cost for completion call with gpt-3.5-turbo: ", f"${float(cost):.10f}")
+```
+
+**Output**
+```shell
+Cost for completion call with gpt-3.5-turbo:  $0.0000775000
+```
+
+### Track Costs, Usage, Latency for streaming
+We use a custom callback function for this - more info on custom callbacks: https://docs.litellm.ai/docs/observability/custom_callback
+- We define a callback function to calculate cost `def track_cost_callback()`
+- In `def track_cost_callback()` we check if the stream is complete - `if "complete_streaming_response" in kwargs`
+- Use `litellm.completion_cost()` to calculate cost, once the stream is complete
+
+```python
+import litellm
+
+# track_cost_callback 
+def track_cost_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    try:
+        # check if it has collected an entire stream response
+        if "complete_streaming_response" in kwargs:
+            # for tracking streaming cost we pass the "messages" and the output_text to litellm.completion_cost 
+            completion_response=kwargs["complete_streaming_response"]
+            input_text = kwargs["messages"]
+            output_text = completion_response["choices"][0]["message"]["content"]
+            response_cost = litellm.completion_cost(
+                model = kwargs["model"],
+                messages = input_text,
+                completion=output_text
+            )
+            print("streaming response_cost", response_cost)
+    except:
+        pass
+# set callback 
+litellm.success_callback = [track_cost_callback] # set custom callback function
+
+# litellm.completion() call
+response = completion(
+    model="gpt-3.5-turbo",
+    messages=[
+        {
+            "role": "user",
+            "content": "Hi 👋 - i'm openai"
+        }
+    ],
+    stream=True
+)
+```
+
+## More details
+* [exception mapping](./exception_mapping.md)
+* [retries + model fallbacks for completion()](./completion/reliable_completions.md)
+* [tutorial for model fallbacks with completion()](./tutorials/fallbacks.md)

docs/my-website/docs/langchain/langchain.md

@@ -0,0 +1,135 @@
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+# Using ChatLiteLLM() - Langchain
+
+## Pre-Requisites
+```shell
+!pip install litellm langchain
+```
+## Quick Start
+
+<Tabs>
+<TabItem value="openai" label="OpenAI">
+
+```python
+import os
+from langchain.chat_models import ChatLiteLLM
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    AIMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import AIMessage, HumanMessage, SystemMessage
+
+os.environ['OPENAI_API_KEY'] = ""
+chat = ChatLiteLLM(model="gpt-3.5-turbo")
+messages = [
+    HumanMessage(
+        content="what model are you"
+    )
+]
+chat(messages)
+```
+
+</TabItem>
+
+<TabItem value="anthropic" label="Anthropic">
+
+```python
+import os
+from langchain.chat_models import ChatLiteLLM
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    AIMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import AIMessage, HumanMessage, SystemMessage
+
+os.environ['ANTHROPIC_API_KEY'] = ""
+chat = ChatLiteLLM(model="claude-2", temperature=0.3)
+messages = [
+    HumanMessage(
+        content="what model are you"
+    )
+]
+chat(messages)
+```
+
+</TabItem>
+
+<TabItem value="replicate" label="Replicate">
+
+```python
+import os
+from langchain.chat_models import ChatLiteLLM
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    AIMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import AIMessage, HumanMessage, SystemMessage
+
+os.environ['REPLICATE_API_TOKEN'] = ""
+chat = ChatLiteLLM(model="replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1")
+messages = [
+    HumanMessage(
+        content="what model are you?"
+    )
+]
+chat(messages)
+```
+
+</TabItem>
+
+<TabItem value="cohere" label="Cohere">
+
+```python
+import os
+from langchain.chat_models import ChatLiteLLM
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    AIMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import AIMessage, HumanMessage, SystemMessage
+
+os.environ['COHERE_API_KEY'] = ""
+chat = ChatLiteLLM(model="command-nightly")
+messages = [
+    HumanMessage(
+        content="what model are you?"
+    )
+]
+chat(messages)
+```
+
+</TabItem>
+<TabItem value="palm" label="PaLM - Google">
+
+```python
+import os
+from langchain.chat_models import ChatLiteLLM
+from langchain.prompts.chat import (
+    ChatPromptTemplate,
+    SystemMessagePromptTemplate,
+    AIMessagePromptTemplate,
+    HumanMessagePromptTemplate,
+)
+from langchain.schema import AIMessage, HumanMessage, SystemMessage
+
+os.environ['PALM_API_KEY'] = ""
+chat = ChatLiteLLM(model="palm/chat-bison")
+messages = [
+    HumanMessage(
+        content="what model are you?"
+    )
+]
+chat(messages)
+```
+</TabItem>
+</Tabs>

docs/my-website/docs/migration.md

@@ -0,0 +1,35 @@
+# Migration Guide - LiteLLM v1.0.0+ 
+
+When we have breaking changes (i.e. going from 1.x.x to 2.x.x), we will document those changes here.
+
+
+## `1.0.0` 
+
+**Last Release before breaking change**: 0.14.0
+
+**What changed?**
+
+- Requires `openai>=1.0.0`
+- `openai.InvalidRequestError` → `openai.BadRequestError`
+- `openai.ServiceUnavailableError` → `openai.APIStatusError`
+- *NEW* litellm client, allow users to pass api_key
+    - `litellm.Litellm(api_key="sk-123")`
+- response objects now inherit from `BaseModel` (prev. `OpenAIObject`)
+- *NEW* default exception - `APIConnectionError` (prev. `APIError`)
+- litellm.get_max_tokens() now returns an int not a dict
+    ```python
+    max_tokens = litellm.get_max_tokens("gpt-3.5-turbo") # returns an int not a dict 
+    assert max_tokens==4097
+    ```
+- Streaming - OpenAI Chunks now return `None` for empty stream chunks. This is how to process stream chunks with content
+    ```python
+    response = litellm.completion(model="gpt-3.5-turbo", messages=messages, stream=True)
+    for part in response:
+        print(part.choices[0].delta.content or "")
+    ```
+
+**How can we communicate changes better?**
+Tell us
+- [Discord](https://discord.com/invite/wuPM9dRgDw)
+- Email ([email protected]/[email protected])
+- Text us (+17708783106)

docs/my-website/docs/observability/callbacks.md

@@ -0,0 +1,35 @@
+# Callbacks
+
+## Use Callbacks to send Output Data to Posthog, Sentry etc
+
+liteLLM provides `input_callbacks`, `success_callbacks` and `failure_callbacks`, making it easy for you to send data to a particular provider depending on the status of your responses.
+
+liteLLM supports:
+
+- [Custom Callback Functions](https://docs.litellm.ai/docs/observability/custom_callback)
+- [LLMonitor](https://llmonitor.com/docs)
+- [Helicone](https://docs.helicone.ai/introduction)
+- [Traceloop](https://traceloop.com/docs)
+- [Sentry](https://docs.sentry.io/platforms/python/)
+- [PostHog](https://posthog.com/docs/libraries/python)
+- [Slack](https://slack.dev/bolt-python/concepts)
+
+### Quick Start
+
+```python
+from litellm import completion
+
+# set callbacks
+litellm.input_callback=["sentry"] # for sentry breadcrumbing - logs the input being sent to the api
+litellm.success_callback=["posthog", "helicone", "llmonitor"]
+litellm.failure_callback=["sentry", "llmonitor"]
+
+## set env variables
+os.environ['SENTRY_DSN'], os.environ['SENTRY_API_TRACE_RATE']= ""
+os.environ['POSTHOG_API_KEY'], os.environ['POSTHOG_API_URL'] = "api-key", "api-url"
+os.environ["HELICONE_API_KEY"] = ""
+os.environ["TRACELOOP_API_KEY"] = ""
+os.environ["LLMONITOR_APP_ID"] = ""
+
+response = completion(model="gpt-3.5-turbo", messages=messages)
+```

docs/my-website/docs/observability/custom_callback.md

@@ -0,0 +1,358 @@
+# Custom Callbacks
+
+## Callback Class
+You can create a custom callback class to precisely log events as they occur in litellm. 
+
+```python
+from litellm.integrations.custom_logger import CustomLogger
+
+class MyCustomHandler(CustomLogger):
+    def log_pre_api_call(self, model, messages, kwargs): 
+        print(f"Pre-API Call")
+    
+    def log_post_api_call(self, kwargs, response_obj, start_time, end_time): 
+        print(f"Post-API Call")
+    
+    def log_stream_event(self, kwargs, response_obj, start_time, end_time):
+        print(f"On Stream")
+        
+    def log_success_event(self, kwargs, response_obj, start_time, end_time): 
+        print(f"On Success")
+
+    def log_failure_event(self, kwargs, response_obj, start_time, end_time): 
+        print(f"On Failure")
+
+customHandler = MyCustomHandler()
+
+litellm.callbacks = [customHandler]
+response = completion(model="gpt-3.5-turbo", messages=[{ "role": "user", "content": "Hi 👋 - i'm openai"}],
+                              stream=True)
+for chunk in response: 
+    continue
+```
+
+## Callback Functions
+If you just want to log on a specific event (e.g. on input) - you can use callback functions. 
+
+You can set custom callbacks to trigger for:
+- `litellm.input_callback`   - Track inputs/transformed inputs before making the LLM API call
+- `litellm.success_callback` - Track inputs/outputs after making LLM API call
+- `litellm.failure_callback` - Track inputs/outputs + exceptions for litellm calls
+
+## Defining a Custom Callback Function
+Create a custom callback function that takes specific arguments:
+
+```python
+def custom_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    # Your custom code here
+    print("LITELLM: in custom callback function")
+    print("kwargs", kwargs)
+    print("completion_response", completion_response)
+    print("start_time", start_time)
+    print("end_time", end_time)
+```
+
+### Setting the custom callback function
+```python
+import litellm
+litellm.success_callback = [custom_callback]
+```
+
+## Using Your Custom Callback Function
+
+```python
+import litellm
+from litellm import completion
+
+# Assign the custom callback function
+litellm.success_callback = [custom_callback]
+
+response = completion(
+    model="gpt-3.5-turbo",
+    messages=[
+        {
+            "role": "user",
+            "content": "Hi 👋 - i'm openai"
+        }
+    ]
+)
+
+print(response)
+
+```
+
+## Async Callback Functions 
+
+LiteLLM currently supports just async success callback functions for async completion/embedding calls. 
+
+```python
+import asyncio, litellm 
+
+async def async_test_logging_fn(kwargs, completion_obj, start_time, end_time):
+    print(f"On Async Success!")
+
+async def test_chat_openai():
+    try:
+        # litellm.set_verbose = True
+        litellm.success_callback = [async_test_logging_fn]
+        response = await litellm.acompletion(model="gpt-3.5-turbo",
+                              messages=[{
+                                  "role": "user",
+                                  "content": "Hi 👋 - i'm openai"
+                              }],
+                              stream=True)
+        async for chunk in response: 
+            continue
+    except Exception as e:
+        print(e)
+        pytest.fail(f"An error occurred - {str(e)}")
+
+asyncio.run(test_chat_openai())
+```
+
+:::info
+
+We're actively trying to expand this to other event types. [Tell us if you need this!](https://github.com/BerriAI/litellm/issues/1007)
+
+
+
+:::
+
+## What's in kwargs? 
+
+Notice we pass in a kwargs argument to custom callback. 
+```python
+def custom_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    # Your custom code here
+    print("LITELLM: in custom callback function")
+    print("kwargs", kwargs)
+    print("completion_response", completion_response)
+    print("start_time", start_time)
+    print("end_time", end_time)
+```
+
+This is a dictionary containing all the model-call details (the params we receive, the values we send to the http endpoint, the response we receive, stacktrace in case of errors, etc.). 
+
+This is all logged in the [model_call_details via our Logger](https://github.com/BerriAI/litellm/blob/fc757dc1b47d2eb9d0ea47d6ad224955b705059d/litellm/utils.py#L246).
+
+Here's exactly what you can expect in the kwargs dictionary:
+```shell
+### DEFAULT PARAMS ### 
+"model": self.model,
+"messages": self.messages,
+"optional_params": self.optional_params, # model-specific params passed in
+"litellm_params": self.litellm_params, # litellm-specific params passed in (e.g. metadata passed to completion call)
+"start_time": self.start_time, # datetime object of when call was started
+
+### PRE-API CALL PARAMS ### (check via kwargs["log_event_type"]="pre_api_call")
+"input" = input # the exact prompt sent to the LLM API
+"api_key" = api_key # the api key used for that LLM API 
+"additional_args" = additional_args # any additional details for that API call (e.g. contains optional params sent)
+
+### POST-API CALL PARAMS ### (check via kwargs["log_event_type"]="post_api_call")
+"original_response" = original_response # the original http response received (saved via response.text)
+
+### ON-SUCCESS PARAMS ### (check via kwargs["log_event_type"]="successful_api_call")
+"complete_streaming_response" = complete_streaming_response # the complete streamed response (only set if `completion(..stream=True)`)
+"end_time" = end_time # datetime object of when call was completed
+
+### ON-FAILURE PARAMS ### (check via kwargs["log_event_type"]="failed_api_call")
+"exception" = exception # the Exception raised
+"traceback_exception" = traceback_exception # the traceback generated via `traceback.format_exc()`
+"end_time" = end_time # datetime object of when call was completed
+```
+
+### Get complete streaming response
+
+LiteLLM will pass you the complete streaming response in the final streaming chunk as part of the kwargs for your custom callback function.
+
+```python
+# litellm.set_verbose = False
+        def custom_callback(
+            kwargs,                 # kwargs to completion
+            completion_response,    # response from completion
+            start_time, end_time    # start/end time
+        ):
+            # print(f"streaming response: {completion_response}")
+            if "complete_streaming_response" in kwargs: 
+                print(f"Complete Streaming Response: {kwargs['complete_streaming_response']}")
+        
+        # Assign the custom callback function
+        litellm.success_callback = [custom_callback]
+
+        response = completion(model="claude-instant-1", messages=messages, stream=True)
+        for idx, chunk in enumerate(response): 
+            pass
+```
+
+
+### Log additional metadata
+
+LiteLLM accepts a metadata dictionary in the completion call. You can pass additional metadata into your completion call via `completion(..., metadata={"key": "value"})`. 
+
+Since this is a [litellm-specific param](https://github.com/BerriAI/litellm/blob/b6a015404eed8a0fa701e98f4581604629300ee3/litellm/main.py#L235), it's accessible via kwargs["litellm_params"]
+
+```python
+from litellm import completion
+import os, litellm
+
+## set ENV variables
+os.environ["OPENAI_API_KEY"] = "your-api-key"
+
+messages = [{ "content": "Hello, how are you?","role": "user"}]
+
+def custom_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    print(kwargs["litellm_params"]["metadata"])
+    
+
+# Assign the custom callback function
+litellm.success_callback = [custom_callback]
+
+response = litellm.completion(model="gpt-3.5-turbo", messages=messages, metadata={"hello": "world"})
+```
+
+## Examples
+
+### Custom Callback to track costs for Streaming + Non-Streaming
+```python
+
+def track_cost_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    try:
+        # init logging config
+        logging.basicConfig(
+                filename='cost.log',
+                level=logging.INFO,
+                format='%(asctime)s - %(message)s',
+                datefmt='%Y-%m-%d %H:%M:%S'
+        )
+
+        # check if it has collected an entire stream response
+        if "complete_streaming_response" in kwargs:
+            # for tracking streaming cost we pass the "messages" and the output_text to litellm.completion_cost 
+            completion_response=kwargs["complete_streaming_response"]
+            input_text = kwargs["messages"]
+            output_text = completion_response["choices"][0]["message"]["content"]
+            response_cost = litellm.completion_cost(
+                model = kwargs["model"],
+                messages = input_text,
+                completion=output_text
+            )
+            print("streaming response_cost", response_cost)
+            logging.info(f"Model {kwargs['model']} Cost: ${response_cost:.8f}")
+
+        # for non streaming responses
+        else:
+            # we pass the completion_response obj
+            if kwargs["stream"] != True:
+                response_cost = litellm.completion_cost(completion_response=completion_response)
+                print("regular response_cost", response_cost)
+                logging.info(f"Model {completion_response.model} Cost: ${response_cost:.8f}")
+    except:
+        pass
+
+# Assign the custom callback function
+litellm.success_callback = [track_cost_callback]
+
+response = completion(
+    model="gpt-3.5-turbo",
+    messages=[
+        {
+            "role": "user",
+            "content": "Hi 👋 - i'm openai"
+        }
+    ]
+)
+
+print(response)
+```
+
+### Custom Callback to log transformed Input to LLMs
+```python
+def get_transformed_inputs(
+    kwargs,
+):
+    params_to_model = kwargs["additional_args"]["complete_input_dict"]
+    print("params to model", params_to_model)
+
+litellm.input_callback = [get_transformed_inputs]
+
+def test_chat_openai():
+    try:
+        response = completion(model="claude-2",
+                              messages=[{
+                                  "role": "user",
+                                  "content": "Hi 👋 - i'm openai"
+                              }])
+
+        print(response)
+
+    except Exception as e:
+        print(e)
+        pass
+```
+
+#### Output
+```shell
+params to model {'model': 'claude-2', 'prompt': "\n\nHuman: Hi 👋 - i'm openai\n\nAssistant: ", 'max_tokens_to_sample': 256}
+```
+
+### Custom Callback to write to Mixpanel
+
+```python
+import mixpanel
+import litellm
+from litellm import completion
+
+def custom_callback(
+    kwargs,                 # kwargs to completion
+    completion_response,    # response from completion
+    start_time, end_time    # start/end time
+):
+    # Your custom code here
+    mixpanel.track("LLM Response", {"llm_response": completion_response})
+
+
+# Assign the custom callback function
+litellm.success_callback = [custom_callback]
+
+response = completion(
+    model="gpt-3.5-turbo",
+    messages=[
+        {
+            "role": "user",
+            "content": "Hi 👋 - i'm openai"
+        }
+    ]
+)
+
+print(response)
+
+```
+
+
+
+
+
+
+
+
+
+
+
+

docs/my-website/docs/observability/helicone_integration.md

@@ -0,0 +1,55 @@
+# Helicone Tutorial 
+[Helicone](https://helicone.ai/) is an open source observability platform that proxies your OpenAI traffic and provides you key insights into your spend, latency and usage.
+
+## Use Helicone to log requests across all LLM Providers (OpenAI, Azure, Anthropic, Cohere, Replicate, PaLM)
+liteLLM provides `success_callbacks` and `failure_callbacks`, making it easy for you to send data to a particular provider depending on the status of your responses. 
+
+In this case, we want to log requests to Helicone when a request succeeds. 
+
+### Approach 1: Use Callbacks 
+Use just 1 line of code, to instantly log your responses **across all providers** with helicone: 
+```python
+litellm.success_callback=["helicone"]
+```
+
+Complete code
+```python
+from litellm import completion
+
+## set env variables
+os.environ["HELICONE_API_KEY"] = "your-helicone-key" 
+os.environ["OPENAI_API_KEY"], os.environ["COHERE_API_KEY"] = "", ""
+
+# set callbacks
+litellm.success_callback=["helicone"]
+
+#openai call
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}]) 
+
+#cohere call
+response = completion(model="command-nightly", messages=[{"role": "user", "content": "Hi 👋 - i'm cohere"}]) 
+```
+
+### Approach 2: [OpenAI + Azure only] Use Helicone as a proxy
+Helicone provides advanced functionality like caching, etc. Helicone currently supports this for Azure and OpenAI.
+
+If you want to use Helicone to proxy your OpenAI/Azure requests, then you can - 
+
+- Set helicone as your base url via: `litellm.api_url` 
+- Pass in helicone request headers via: `litellm.headers` 
+
+Complete Code
+```python
+import litellm
+from litellm import completion
+
+litellm.api_base = "https://oai.hconeai.com/v1"
+litellm.headers = {"Helicone-Auth": f"Bearer {os.getenv('HELICONE_API_KEY')}"}
+
+response = litellm.completion(
+    model="gpt-3.5-turbo",
+    messages=[{"role": "user", "content": "how does a court case get to the Supreme Court?"}]
+)
+
+print(response)
+```

docs/my-website/docs/observability/langfuse_integration.md

@@ -0,0 +1,105 @@
+import Image from '@theme/IdealImage';
+
+# Langfuse - Logging LLM Input/Output
+
+LangFuse is open Source Observability & Analytics for LLM Apps
+Detailed production traces and a granular view on quality, cost and latency
+
+<Image img={require('../../img/langfuse.png')} />
+
+:::info
+We want to learn how we can make the callbacks better! Meet the LiteLLM [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
+join our [discord](https://discord.gg/wuPM9dRgDw)
+::: 
+
+## Pre-Requisites
+Ensure you have run `pip install langfuse` for this integration
+```shell
+pip install langfuse litellm
+```
+
+## Quick Start
+Use just 2 lines of code, to instantly log your responses **across all providers** with Langfuse
+<a target="_blank" href="https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/logging_observability/LiteLLM_Langfuse.ipynb">
+  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
+</a>
+
+Get your Langfuse API Keys from https://cloud.langfuse.com/
+```python
+litellm.success_callback = ["langfuse"]
+```
+```python
+# pip install langfuse 
+import litellm
+import os
+
+# from https://cloud.langfuse.com/
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+# Optional, defaults to https://cloud.langfuse.com
+os.environ["LANGFUSE_HOST"] # optional
+
+# LLM API Keys
+os.environ['OPENAI_API_KEY']=""
+
+# set langfuse as a callback, litellm will send the data to langfuse
+litellm.success_callback = ["langfuse"] 
+ 
+# openai call
+response = litellm.completion(
+  model="gpt-3.5-turbo",
+  messages=[
+    {"role": "user", "content": "Hi 👋 - i'm openai"}
+  ]
+)
+```
+
+## Advanced
+### Set Custom Generation names, pass metadata
+
+```python
+import litellm
+from litellm import completion
+import os
+
+# from https://cloud.langfuse.com/
+os.environ["LANGFUSE_PUBLIC_KEY"] = ""
+os.environ["LANGFUSE_SECRET_KEY"] = ""
+
+
+# OpenAI and Cohere keys 
+# You can use any of the litellm supported providers: https://docs.litellm.ai/docs/providers
+os.environ['OPENAI_API_KEY']=""
+
+# set langfuse as a callback, litellm will send the data to langfuse
+litellm.success_callback = ["langfuse"] 
+ 
+# openai call
+response = completion(
+  model="gpt-3.5-turbo",
+  messages=[
+    {"role": "user", "content": "Hi 👋 - i'm openai"}
+  ],
+  metadata = {
+    "generation_name": "litellm-ishaan-gen", # set langfuse generation name
+    # custom metadata fields
+    "project": "litellm-proxy" 
+  }
+)
+ 
+print(response)
+
+```
+
+
+
+## Troubleshooting & Errors
+### Data not getting logged to Langfuse ? 
+- Ensure you're on the latest version of langfuse `pip install langfuse -U`. The latest version allows litellm to log JSON input/outputs to langfuse
+
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/observability/langsmith_integration.md

@@ -0,0 +1,77 @@
+import Image from '@theme/IdealImage';
+
+# Langsmith - Logging LLM Input/Output
+An all-in-one developer platform for every step of the application lifecycle
+https://smith.langchain.com/
+
+<Image img={require('../../img/langsmith.png')} />
+
+:::info
+We want to learn how we can make the callbacks better! Meet the LiteLLM [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
+join our [discord](https://discord.gg/wuPM9dRgDw)
+::: 
+
+## Pre-Requisites
+```shell
+pip install litellm
+```
+
+## Quick Start
+Use just 2 lines of code, to instantly log your responses **across all providers** with Langsmith
+
+
+```python
+litellm.success_callback = ["langsmith"]
+```
+```python
+import litellm
+import os
+
+os.environ["LANGSMITH_API_KEY"] = ""
+# LLM API Keys
+os.environ['OPENAI_API_KEY']=""
+
+# set langsmith as a callback, litellm will send the data to langsmith
+litellm.success_callback = ["langsmith"] 
+ 
+# openai call
+response = litellm.completion(
+  model="gpt-3.5-turbo",
+  messages=[
+    {"role": "user", "content": "Hi 👋 - i'm openai"}
+  ]
+)
+```
+
+## Advanced
+### Set Custom Project & Run names
+
+```python
+import litellm
+import os
+
+os.environ["LANGSMITH_API_KEY"] = ""
+# LLM API Keys
+os.environ['OPENAI_API_KEY']=""
+
+# set langfuse as a callback, litellm will send the data to langfuse
+litellm.success_callback = ["langfuse"] 
+ 
+response = litellm.completion(
+    model="gpt-3.5-turbo",
+     messages=[
+        {"role": "user", "content": "Hi 👋 - i'm openai"}
+    ],
+    metadata={
+        "run_name": "litellmRUN",               # langsmith run name
+        "project_name": "litellm-completion",   # langsmith project name
+    }
+)
+print(response)
+```
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/observability/llmonitor_integration.md

@@ -0,0 +1,65 @@
+# LLMonitor Tutorial
+
+[LLMonitor](https://llmonitor.com/) is an open-source observability platform that provides cost tracking, user tracking and powerful agent tracing.
+
+<video controls width='900' >
+  <source src='https://llmonitor.com/videos/demo-annotated.mp4'/>
+</video>
+
+## Use LLMonitor to log requests across all LLM Providers (OpenAI, Azure, Anthropic, Cohere, Replicate, PaLM)
+
+liteLLM provides `callbacks`, making it easy for you to log data depending on the status of your responses.
+
+:::info
+We want to learn how we can make the callbacks better! Meet the [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
+join our [discord](https://discord.gg/wuPM9dRgDw)
+::: 
+
+### Using Callbacks
+
+First, sign up to get an app ID on the [LLMonitor dashboard](https://llmonitor.com).
+
+Use just 2 lines of code, to instantly log your responses **across all providers** with llmonitor:
+
+```python
+litellm.success_callback = ["llmonitor"]
+litellm.failure_callback = ["llmonitor"]
+```
+
+Complete code
+
+```python
+from litellm import completion
+
+## set env variables
+os.environ["LLMONITOR_APP_ID"] = "your-llmonitor-app-id"
+# Optional: os.environ["LLMONITOR_API_URL"] = "self-hosting-url"
+
+os.environ["OPENAI_API_KEY"], os.environ["COHERE_API_KEY"] = "", ""
+
+# set callbacks
+litellm.success_callback = ["llmonitor"]
+litellm.failure_callback = ["llmonitor"]
+
+#openai call
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}],
+  user="ishaan_litellm"
+)
+
+#cohere call
+response = completion(
+  model="command-nightly", 
+  messages=[{"role": "user", "content": "Hi 👋 - i'm cohere"}],
+  user="ishaan_litellm"
+)
+```
+
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]
+- Meet the LLMonitor team on [Discord](http://discord.com/invite/8PafSG58kK) or via [email](mailto:[email protected]).

docs/my-website/docs/observability/promptlayer_integration.md

@@ -0,0 +1,77 @@
+# Promptlayer Tutorial
+
+Promptlayer is a platform for prompt engineers. Log OpenAI requests. Search usage history. Track performance. Visually manage prompt templates.
+
+<Image img={require('../../img/promptlayer.png')} />
+
+## Use Promptlayer to log requests across all LLM Providers (OpenAI, Azure, Anthropic, Cohere, Replicate, PaLM)
+
+liteLLM provides `callbacks`, making it easy for you to log data depending on the status of your responses.
+
+### Using Callbacks
+
+Get your PromptLayer API Key from https://promptlayer.com/
+
+Use just 2 lines of code, to instantly log your responses **across all providers** with promptlayer:
+
+```python
+litellm.success_callback = ["promptlayer"]
+
+```
+
+Complete code
+
+```python
+from litellm import completion
+
+## set env variables
+os.environ["PROMPTLAYER_API_KEY"] = "your-promptlayer-key"
+
+os.environ["OPENAI_API_KEY"], os.environ["COHERE_API_KEY"] = "", ""
+
+# set callbacks
+litellm.success_callback = ["promptlayer"]
+
+#openai call
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}])
+
+#cohere call
+response = completion(model="command-nightly", messages=[{"role": "user", "content": "Hi 👋 - i'm cohere"}])
+```
+
+### Logging Metadata 
+
+You can also log completion call metadata to Promptlayer. 
+
+You can add metadata to a completion call through the metadata param: 
+```python 
+completion(model,messages, metadata={"model": "ai21"})
+```
+
+**Complete Code**
+```python
+from litellm import completion
+
+## set env variables
+os.environ["PROMPTLAYER_API_KEY"] = "your-promptlayer-key"
+
+os.environ["OPENAI_API_KEY"], os.environ["COHERE_API_KEY"] = "", ""
+
+# set callbacks
+litellm.success_callback = ["promptlayer"]
+
+#openai call - log llm provider is openai
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}], metadata={"provider": "openai"})
+
+#cohere call - log llm provider is cohere
+response = completion(model="command-nightly", messages=[{"role": "user", "content": "Hi 👋 - i'm cohere"}], metadata={"provider": "cohere"})
+```
+
+Credits to [Nick Bradford](https://github.com/nsbradford), from [Vim-GPT](https://github.com/nsbradford/VimGPT), for the suggestion. 
+
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/observability/sentry.md

@@ -0,0 +1,44 @@
+import Image from '@theme/IdealImage';
+
+# Sentry - Log LLM Exceptions
+[Sentry](https://sentry.io/) provides error monitoring for production. LiteLLM can add breadcrumbs and send exceptions to Sentry with this integration
+
+Track exceptions for:
+- litellm.completion() - completion()for 100+ LLMs
+- litellm.acompletion() - async completion()
+- Streaming completion() & acompletion() calls
+
+<Image img={require('../../img/sentry.png')} />
+
+
+## Usage
+
+### Set SENTRY_DSN & callback
+
+```python
+import litellm, os
+os.environ["SENTRY_DSN"] = "your-sentry-url"
+litellm.failure_callback=["sentry"]
+```
+
+### Sentry callback with completion
+```python
+import litellm
+from litellm import completion 
+
+litellm.input_callback=["sentry"] # adds sentry breadcrumbing
+litellm.failure_callback=["sentry"] # [OPTIONAL] if you want litellm to capture -> send exception to sentry
+
+import os 
+os.environ["SENTRY_DSN"] = "your-sentry-url"
+os.environ["OPENAI_API_KEY"] = "your-openai-key"
+
+# set bad key to trigger error 
+api_key="bad-key"
+response = completion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey!"}], stream=True, api_key=api_key)
+
+print(response)
+```
+
+[Let us know](https://github.com/BerriAI/litellm/issues/new?assignees=&labels=enhancement&projects=&template=feature_request.yml&title=%5BFeature%5D%3A+) if you need any additional options from Sentry. 
+

docs/my-website/docs/observability/slack_integration.md

@@ -0,0 +1,93 @@
+import Image from '@theme/IdealImage';
+
+# Slack - Logging LLM Input/Output, Exceptions
+
+<Image img={require('../../img/slack.png')} />
+
+:::info
+We want to learn how we can make the callbacks better! Meet the LiteLLM [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
+join our [discord](https://discord.gg/wuPM9dRgDw)
+::: 
+
+## Pre-Requisites
+
+### Step 1
+```shell
+pip install litellm
+```
+
+### Step 2
+Get a slack webhook url from https://api.slack.com/messaging/webhooks
+
+
+
+## Quick Start
+### Create a custom Callback to log to slack
+We create a custom callback, to log to slack webhooks, see [custom callbacks on litellm](https://docs.litellm.ai/docs/observability/custom_callback)
+```python
+def send_slack_alert(
+        kwargs,
+        completion_response,
+        start_time,
+        end_time,
+):
+    print(
+        "in custom slack callback func"
+    )
+    import requests
+    import json
+
+    # Define the Slack webhook URL
+    # get it from https://api.slack.com/messaging/webhooks
+    slack_webhook_url = os.environ['SLACK_WEBHOOK_URL']   # "https://hooks.slack.com/services/<>/<>/<>"
+
+    # Define the text payload, send data available in litellm custom_callbacks
+    text_payload = f"""LiteLLM Logging: kwargs: {str(kwargs)}\n\n, response: {str(completion_response)}\n\n, start time{str(start_time)} end time: {str(end_time)}
+    """
+    payload = {
+        "text": text_payload
+    }
+
+    # Set the headers
+    headers = {
+        "Content-type": "application/json"
+    }
+
+    # Make the POST request
+    response = requests.post(slack_webhook_url, json=payload, headers=headers)
+
+    # Check the response status
+    if response.status_code == 200:
+        print("Message sent successfully to Slack!")
+    else:
+        print(f"Failed to send message to Slack. Status code: {response.status_code}")
+        print(response.json())
+```
+
+### Pass callback to LiteLLM
+```python
+litellm.success_callback = [send_slack_alert]
+```
+
+```python
+import litellm
+litellm.success_callback = [send_slack_alert] # log success
+litellm.failure_callback = [send_slack_alert] # log exceptions
+
+# this will raise an exception
+response = litellm.completion(
+    model="gpt-2",
+    messages=[
+        {
+            "role": "user",
+            "content": "Hi 👋 - i'm openai"
+        }
+    ]
+)
+```
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/observability/supabase_integration.md

@@ -0,0 +1,101 @@
+# Supabase Tutorial 
+[Supabase](https://supabase.com/) is an open source Firebase alternative.
+Start your project with a Postgres database, Authentication, instant APIs, Edge Functions, Realtime subscriptions, Storage, and Vector embeddings.
+
+## Use Supabase to log requests and see total spend across all LLM Providers (OpenAI, Azure, Anthropic, Cohere, Replicate, PaLM)
+liteLLM provides `success_callbacks` and `failure_callbacks`, making it easy for you to send data to a particular provider depending on the status of your responses. 
+
+In this case, we want to log requests to Supabase in both scenarios - when it succeeds and fails. 
+
+### Create a supabase table 
+
+Go to your Supabase project > go to the [Supabase SQL Editor](https://supabase.com/dashboard/projects) and create a new table with this configuration.
+
+Note: You can change the table name. Just don't change the column names. 
+
+```sql
+create table
+  public.request_logs (
+    id bigint generated by default as identity,
+    created_at timestamp with time zone null default now(),
+    model text null default ''::text,
+    messages json null default '{}'::json,
+    response json null default '{}'::json,
+    end_user text null default ''::text,
+    status text null default ''::text,
+    error json null default '{}'::json,
+    response_time real null default '0'::real,
+    total_cost real null,
+    additional_details json null default '{}'::json,
+    litellm_call_id text unique,
+    primary key (id)
+  ) tablespace pg_default;
+```
+
+### Use Callbacks 
+Use just 2 lines of code, to instantly see costs and log your responses **across all providers** with Supabase: 
+
+```python
+litellm.success_callback=["supabase"]
+litellm.failure_callback=["supabase"]
+```
+
+Complete code
+```python
+from litellm import completion
+
+## set env variables
+### SUPABASE
+os.environ["SUPABASE_URL"] = "your-supabase-url" 
+os.environ["SUPABASE_KEY"] = "your-supabase-key" 
+
+## LLM API KEY
+os.environ["OPENAI_API_KEY"] = ""
+
+# set callbacks
+litellm.success_callback=["supabase"]
+litellm.failure_callback=["supabase"]
+
+# openai call
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}],
+  user="ishaan22" # identify users
+) 
+
+# bad call, expect this call to fail and get logged
+response = completion(
+  model="chatgpt-test", 
+  messages=[{"role": "user", "content": "Hi 👋 - i'm a bad call to test error logging"}]
+)
+ 
+```
+
+### Additional Controls 
+
+**Identify end-user**
+
+Pass `user` to `litellm.completion` to map your llm call to an end-user 
+
+```python
+response = completion(
+  model="gpt-3.5-turbo", 
+  messages=[{"role": "user", "content": "Hi 👋 - i'm openai"}],
+  user="ishaan22" # identify users
+) 
+```
+
+**Different Table name**
+
+If you modified your table name, here's how to pass the new name.
+
+```python 
+litellm.modify_integration("supabase",{"table_name": "litellm_logs"})
+```
+
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/observability/telemetry.md

@@ -0,0 +1,13 @@
+# Telemetry 
+
+LiteLLM contains a telemetry feature that tells us what models are used, and what errors are hit.
+
+## What is logged? 
+
+Only the model name and exception raised is logged. 
+
+## Why?
+We use this information to help us understand how LiteLLM is used, and improve stability. 
+
+## Opting out
+If you prefer to opt out of telemetry, you can do this by setting `litellm.telemetry = False`. 

docs/my-website/docs/observability/traceloop_integration.md

@@ -0,0 +1,34 @@
+import Image from '@theme/IdealImage';
+
+# Traceloop (OpenLLMetry) - Tracing LLMs with OpenTelemetry
+
+[Traceloop](https://traceloop.com) is a platform for monitoring and debugging the quality of your LLM outputs.
+It provides you with a way to track the performance of your LLM application; rollout changes with confidence; and debug issues in production.
+It is based on [OpenTelemetry](https://opentelemetry.io), so it can provide full visibility to your LLM requests, as well vector DB usage, and other infra in your stack.
+
+<Image img={require('../../img/traceloop_dash.png')} />
+
+## Getting Started
+
+Install the Traceloop SDK:
+
+```
+pip install traceloop-sdk
+```
+
+Use just 2 lines of code, to instantly log your LLM responses with OpenTelemetry:
+
+```python
+Traceloop.init(app_name=<YOUR APP NAME>, disable_batch=True)
+litellm.success_callback = ["traceloop"]
+```
+
+To get better visualizations on how your code behaves, you may want to annotate specific parts of your LLM chain. See [Traceloop docs on decorators](https://traceloop.com/docs/python-sdk/decorators) for more information.
+
+## Exporting traces to other systems (e.g. Datadog, New Relic, and others)
+
+Since Traceloop SDK uses OpenTelemetry to send data, you can easily export your traces to other systems, such as Datadog, New Relic, and others. See [Traceloop docs on exporters](https://traceloop.com/docs/python-sdk/exporters) for more information.
+
+## Support
+
+For any question or issue with integration you can reach out to the Traceloop team on [Slack](https://join.slack.com/t/traceloopcommunity/shared_invite/zt-1plpfpm6r-zOHKI028VkpcWdobX65C~g) or via [email](mailto:[email protected]).

docs/my-website/docs/observability/wandb_integration.md

@@ -0,0 +1,51 @@
+import Image from '@theme/IdealImage';
+
+# Weights & Biases - Logging LLM Input/Output
+Weights & Biases helps AI developers build better models faster https://wandb.ai
+
+<Image img={require('../../img/wandb.png')} />
+
+:::info
+We want to learn how we can make the callbacks better! Meet the LiteLLM [founders](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version) or
+join our [discord](https://discord.gg/wuPM9dRgDw)
+::: 
+
+## Pre-Requisites
+Ensure you have run `pip install wandb` for this integration
+```shell
+pip install wandb litellm
+```
+
+## Quick Start
+Use just 2 lines of code, to instantly log your responses **across all providers** with Weights & Biases
+
+```python
+litellm.success_callback = ["wandb"]
+```
+```python
+# pip install wandb 
+import litellm
+import os
+
+os.environ["WANDB_API_KEY"] = ""
+# LLM API Keys
+os.environ['OPENAI_API_KEY']=""
+
+# set wandb as a callback, litellm will send the data to Weights & Biases
+litellm.success_callback = ["wandb"] 
+ 
+# openai call
+response = litellm.completion(
+  model="gpt-3.5-turbo",
+  messages=[
+    {"role": "user", "content": "Hi 👋 - i'm openai"}
+  ]
+)
+```
+
+## Support & Talk to Founders
+
+- [Schedule Demo 👋](https://calendly.com/d/4mp-gd3-k5k/berriai-1-1-onboarding-litellm-hosted-version)
+- [Community Discord 💭](https://discord.gg/wuPM9dRgDw)
+- Our numbers 📞 +1 (770) 8783-106 / ‭+1 (412) 618-6238‬
+- Our emails ✉️ [email protected] / [email protected]

docs/my-website/docs/projects.md

@@ -0,0 +1,19 @@
+# Projects Built on LiteLLM
+
+
+
+### EntoAI
+Chat and Ask on your own data.
+[Github](https://github.com/akshata29/entaoai)
+
+### GPT-Migrate
+Easily migrate your codebase from one framework or language to another.
+[Github](https://github.com/0xpayne/gpt-migrate)
+
+### Otter
+Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context learning ability.
+[Github](https://github.com/Luodian/Otter)
+
+
+
+