Hi All, I will be implementing the huggingface model with gradio app. First, I will download the huggingface model using python script and then use the same model in gradio.
Download the model
Here is the python script to download the huggingface model
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
from huggingface_hub import hf_hub_download
# load the hugging face mode
HUGGINGFACEHUB_API_TOKEN = "<token>"
# download model files
def download_model(model_id):
filenames = [ "pytorch_model.bin", "added_tokens.json", "config.json", "generation_config.json",
"special_tokens_map.json", "spiece.model", "tokenizer_config.json"]
for filename in filenames:
model_path = hf_hub_download(
repo_id=model_id,
filename=filename,
token="HUGGINGFACEHUB_API_TOKEN"
)
print("Downloaded model file:", model_path)
# main
if __name__=='__main__':
model_id = "lmsys/fastchat-t5-3b-v1.0"
download_model(model_id)
Above model will be downloaded into the ~/.cache folder by default, HF will use the model from this cache dirctory.
Usage
Here is the sample python code for running the model with gradio.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
from langchain.llms import HuggingFacePipeline
from langchain import PromptTemplate, LLMChain
import gradio as gr
model_id = "lmsys/fastchat-t5-3b-v1.0"
llm = HuggingFacePipeline.from_model_id(model_id=model_id, task="text2text-generation",
model_kwargs={"temperature": 0, "max_length": 1000})
# define template
template = """
You are a friendly chatbot assistant that responds conversationally to users' questions.
Keep the answers short, unless specifically asked by the user to elaborate on something.
Question: {question}
Answer:"""
# create prompt
prompt = PromptTemplate(template=template, input_variables=["question"])
# llm chain
llm_chain = LLMChain(prompt=prompt,llm=llm)
# ask function
def ask_question(question):
result = llm_chain(question)
return result['text']
iface = gr.Interface(ask_question, inputs=gr.Textbox(lines=2, placeholder="Question Here..."), outputs="text")
iface.launch(server_port=8081)
Now execute the script to start the gradio app, you can also publish it publically by updating launch(share=true)
1
gradio app.py
you can see the gradio app running on the url: http://127.0.0.1:8081, you can ask any question and you will see the results like below
Thanks and Happy Learning !!