Chatgpt的出现给NLP领域带来了让人振奋的消息，可以很逼真的模拟人的对话，回答人们提出的问题，不过Chatgpt参数量，规模，训练代价都很昂贵。
幸运的是，出现了开源的一些相对小的模型，可以在本地或者云端部署体验，
动手体验了下Vicuna-7b，翻译过来是小羊驼」（骆马），拥有70亿参数，据作者实验能达到GPT-4的90%性能。在作者官网发布了三个版本，其中3个月前发布了v1.1，17天前发布了v1.3。
官网：lmsys (Large Model Systems Organization)
环境：
ubuntu18.04
九天毕昇8核32G内存，仅有cpu
时间：2023年7月5号
llama-7B
python3 -m cli --model-path /root/transformer_model_7b --device cpu文件：cli是fastchat/serve目录下的cli.py文件
cli.py内容
"""
Chat with a model with command line interface.
Usage:
python3 -m fastchat.serve.cli --model lmsys/vicuna-7b-v1.3
python3 -m fastchat.serve.cli --model lmsys/fastchat-t5-3b-v1.0
Other commands:
- Type "!!exit" or an empty line to exit.
- Type "!!reset" to start a new conversation.
"""
import argparse
import os
import re
import sys
from prompt_toolkit import PromptSession
from prompt_toolkit.auto_suggest import AutoSuggestFromHistory
from prompt_toolkit.completion import WordCompleter
from prompt_toolkit.history import InMemoryHistory
from prompt_toolkit.key_binding import KeyBindings
from rich.console import Console
from rich.live import Live
from rich.markdown import Markdown
from fastchat.model.model_adapter import add_model_args
from fastchat.modules.gptq import GptqConfig
from fastchat.serve.inference import ChatIO, chat_loop
class SimpleChatIO(ChatIO):
def prompt_for_input(self, role) -> str:
return input(f"{role}: ")
参数： --model-path 是指模型权重存放路径，这个目录下包含tokenizer.model文件，请仔细核对
参数： --device cpu 指定在cpu上运行
开始推理： 输入 who is Musk? what is baidu? china
要等待几分钟，看到几个词几个词的蹦出来
看回答，字数还不少，内容只能说是相关
看下cpu使用率
top可以看到python3进程，推理时cpu使用率在81.1%，一直在波动。
换用v100有时候会到98%，甚至更高，速度会快很多，一次输出一小段文字
vicuna-7b-delta-v1.1
python3 -m cli --model-path /root/vicuna-7b/vicuna-7b-delta-v11 --device cpu加载模型，中间过程要几分钟
python3 -m cli --model-path /root/vicuna-7b/vicuna-7b-delta-v11 --device cpu
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████| 2/2 [02:58<00:00, 89.16s/it]
开始推理：实现一个python斐波那契函数，可以看到能够写出函数，还简单介绍了怎么用
USER: hello
ASSISTANT: Hello! How can I help you today? Is there something you would like to know or talk about? I'm here to help answer any questions you may have.
USER: implement a python function to compute fibonacci numbers
ASSISTANT: Sure! Here's an implementation of a Python function that computes the Fibonacci sequence:
```python
def fibonacci(n):
if n <= 1:
return n
else:
return fibonacci(n-1) + fibonacci(n-2)
```
This function takes an integer `n` as input and returns the `n`-th Fibonacci number. The first two numbers in the sequence are 0 and 1, so the function handles these cases separately: if `n` is less than or equal to 1, it returns `n`, and otherwise it computes the sum of the previous two Fibonacci numbers and returns that.
To use the function, you can call it with an integer `n` and it will return the `n`-th Fibonacci number. For example:
```python
>>> fibonacci(5)
13
```
This would compute the 5-th Fibonacci number, which is 13.
USER: china
ASSISTANT: China is a country located in East Asia, bordered by several other countries, including Russia, Mongolia, North Korea, Vietnam, Laos, Myanmar (Burma), Bhutan, India, Nepal, and Pakistan. It is the world's most populous country, with over 1.4 billion people.
China has a long and complex history, dating back over 5,000 years. It is known for its rich cultural heritage, including its cuisine, literature, and arts. The country has a diverse range of landscapes, from the high plateaus in the west to the fertile river valleys in the east, and from the tropical beaches in the south to the frigid deserts in the north.
vicuna-7b-v1.3
python3 -m cli --model-path /root//vicuna-7b-v13 --device cpu参数：-m 后面指定cli.py文件，注意【.py】省略。python3 -m cli是python3执行cli.py文件，具体内容，要看cli.py写了什么
加载模型，等待几分钟
python3 -m cli --model-path /root//vicuna-7b-v13 --device cpu
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████| 2/2 [03:17<00:00, 98.73s/it]
推理：
USER: Introduce China
ASSISTANT: China, officially known as the People's Republic of China (PRC), is a sovereign state located in East Asia. It is the world's most populous country, with a population of over 1.4 billion people, and the third-largest country by land area, covering approximately 9.6 million square kilometers.
China has a long and complex history, dating back more than 5,000 years. It has been ruled by various dynasties, including the Zhou, Qin, Han, Tang, Song, Yuan, Ming, and Qing dynasties, among others.
The People's Republic of China is a single-party socialist state with a planned economy, guided by the CPC. The country is governed according to the Constitution of the People's Republic of China, which was adopted in 1982. The government is divided into various branches, including the National People's Congress (NPC), the State Council, the Supreme People's Court, and the Central Military Commission.
China has a diverse landscape and culture, encompassing various ethnic groups, including the Han, Tibetans, Uighurs, Mongols, and Manchus, among others. The official language is Standard Chinese (Mandarin), although there are numerous regional languages and dialects spoken throughout the country.
The People's Republic of China is a single-party socialist state with a planned economy, guided by the CPC. The country is governed according to the Constitution of the People's Republic of China, which was adopted in 1982. The government is divided into various branches, including the National People's Congress (NPC), the State Council, the Supreme People's Court, and the Central Military Commission.
China has a diverse landscape and culture, encompassing various ethnic groups, including the Han, Tibetans, Uighurs, Mongols, and Manchus, among others. The official language is Standard Chinese (Mandarin), although there are numerous regional languages and dialects spoken throughout the country.
The economy of China has experienced rapid growth in recent decades, making it the world's second-largest economy by nominal GDP and the largest by purchasing power parity (PPP). The country is a leading global exporter of goods and services, with industries such as electronics, textiles, automobiles, and telecommunications playing a significant role in its economic development.
China has been actively involved in international affairs, participating as a permanent member of the United Nations Security Council and a member of various multilateral organizations, including the World Trade Organization (WTO), the Asia-Pacific Economic Cooperation (APEC), and the Group of 20 (G20) major economies.
USER: china
ASSISTANT: China is a country located in East Asia, covering approximately 9.6 million square kilometers. With a population of over 1.4 billion people, it is the most populous country in the world. The official language is Mandarin Chinese, and the capital city is Beijing. China has a rich history and a diverse cultural heritage, with many different ethnic groups and over 56 recognized ethnic minorities.
目前尚不能支持中文提问和中文回答文章来源：https://www.toymoban.com/news/detail-541434.html
提问方式很重要：输入china，回答的只能说是相关，文字就少很多。输入 Introduce China，输出大段的介绍文章来源地址https://www.toymoban.com/news/detail-541434.html
到了这里，关于试运行llama-7B、vicuna-7b-delta-v1.1和vicuna-7b-v1.3的文章就介绍完了。如果您还想了解更多内容，请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章，希望大家以后多多支持TOY模板网！
![[llama懒人包]ChatGPT本地下位替代llama-7b，支持全平台显卡/CPU运行](https://imgs.yssmx.com/Uploads/2024/02/521746-1.png)
