{"id":1277,"date":"2024-05-01T09:34:06","date_gmt":"2024-05-01T09:34:06","guid":{"rendered":"https:\/\/www.nicekj.com\/?p=1277"},"modified":"2024-05-01T09:34:34","modified_gmt":"2024-05-01T09:34:34","slug":"chatglm26bmoxingzai9ntritonzhongbushubingjichengzhilangchainshijianjingdongyunjishutuandui","status":"publish","type":"post","link":"https:\/\/www.nicekj.com\/chatglm26bmoxingzai9ntritonzhongbushubingjichengzhilangchainshijianjingdongyunjishutuandui.html","title":{"rendered":"chatglm2-6b\u6a21\u578b\u57289n-triton\u4e2d\u90e8\u7f72\u5e76\u96c6\u6210\u81f3langchain\u5b9e\u8df5 | \u4eac\u4e1c\u4e91\u6280\u672f\u56e2\u961f"},"content":{"rendered":"

\u4e00.\u524d\u8a00<\/h1>\n

\u8fd1\u671f\uff0c ChatGLM-6B \u7684\u7b2c\u4e8c\u4ee3\u7248\u672cChatGLM2-6B\u5df2\u7ecf\u6b63\u5f0f\u53d1\u5e03\uff0c\u5f15\u5165\u4e86\u5982\u4e0b\u65b0\u7279\u6027\uff1a<\/p>\n

\u2460. \u57fa\u5ea7\u6a21\u578b\u5347\u7ea7\uff0c\u6027\u80fd\u66f4\u5f3a\u5927\uff0c\u5728\u4e2d\u6587C-Eval\u699c\u5355\u4e2d\uff0c\u4ee551.7\u5206\u4f4d\u5217\u7b2c6\uff1b<\/p>\n

\u2461. \u652f\u63018K-32k\u7684\u4e0a\u4e0b\u6587\uff1b<\/p>\n

\u2462. \u63a8\u7406\u6027\u80fd\u63d0\u5347\u4e8642%\uff1b<\/p>\n

\u2463. \u5bf9\u5b66\u672f\u7814\u7a76\u5b8c\u5168\u5f00\u653e\uff0c\u5141\u8bb8\u7533\u8bf7\u5546\u7528\u6388\u6743\u3002<\/p>\n

\u76ee\u524d\u5927\u591a\u6570\u90e8\u7f72\u65b9\u6848\u91c7\u7528\u7684\u662ffastapi+uvicorn+transformers\uff0c\u8fd9\u79cd\u65b9\u5f0f\u9002\u5408\u5feb\u901f\u8fd0\u884c\u4e00\u4e9bdemo\uff0c\u5728\u751f\u4ea7\u73af\u5883\u4e2d\u4f7f\u7528\u8fd8\u662f\u63a8\u8350\u4f7f\u7528\u4e13\u95e8\u7684\u6df1\u5ea6\u5b66\u4e60\u63a8\u7406\u670d\u52a1\u6846\u67b6\uff0c\u5982Triton\u3002\u672c\u6587\u5c06\u4ecb\u7ecd\u6211\u5229\u7528\u96c6\u56e29n-triton\u5de5\u5177\u90e8\u7f72ChatGLM2-6B\u8fc7\u7a0b\u4e2d\u8e29\u8fc7\u7684\u4e00\u4e9b\u5751\uff0c\u5e0c\u671b\u53ef\u4ee5\u4e3a\u6709\u90e8\u7f72\u9700\u6c42\u7684\u540c\u5b66\u63d0\u4f9b\u4e00\u4e9b\u5e2e\u52a9\u3002<\/p>\n

\u4e8c.\u786c\u4ef6\u8981\u6c42<\/h1>\n

\u90e8\u7f72\u7684\u786c\u4ef6\u8981\u6c42\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\uff1a<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u91cf\u5316\u7b49\u7ea7<\/th>\u7f16\u7801 2048 \u957f\u5ea6\u7684\u6700\u5c0f\u663e\u5b58<\/th>\u751f\u6210 8192 \u957f\u5ea6\u7684\u6700\u5c0f\u663e\u5b58<\/th><\/tr><\/thead>
FP16 \/ BF16<\/td>13.1 GB<\/td>12.8 GB<\/td><\/tr>
INT8<\/td>8.2 GB<\/td>8.1 GB<\/td><\/tr>
INT4<\/td>5.5 GB<\/td>5.1 GB<\/td><\/tr><\/tbody><\/table>\n

\u6211\u90e8\u7f72\u4e862\u4e2apod\uff0c\u6bcf\u4e2apod\u7684\u8d44\u6e90\uff1aCPU\uff084\u6838\uff09\u3001\u5185\u5b58\uff0830G\uff09\u30011\u5f20P40\u663e\u5361\uff08\u663e\u5b5824G\uff09\u3002<\/p>\n

\u4e09.\u90e8\u7f72\u5b9e\u8df5<\/h1>\n

Triton\u9ed8\u8ba4\u652f\u6301\u7684PyTorch\u6a21\u578b\u683c\u5f0f\u4e3aTorchScript\uff0c\u7531\u4e8eChatGLM2-6B\u6a21\u578b\u8f6c\u6362\u6210TorchScript\u683c\u5f0f\u4f1a\u62a5\u9519\uff0c\u672c\u6587\u5c06\u4ee5Python Backend\u7684\u65b9\u5f0f\u8fdb\u884c\u90e8\u7f72\u3002<\/p>\n

1. \u6a21\u578b\u76ee\u5f55\u7ed3\u6784<\/h2>\n

\n

\"chatglm2-6b\u6a21\u578b\u57289n-triton\u4e2d\u90e8\u7f72\u5e76\u96c6\u6210\u81f3langchain\u5b9e\u8df5<\/figure>\n<\/p>\n

9N-Triton\u4f7f\u7528\u96c6\u6210\u6a21\u578b\uff0c\u5982\u4e0a\u56fe\u6240\u793a\u6a21\u578b\u4ed3\u5e93(model_repository), \u5b83\u5185\u90e8\u53ef\u4ee5\u5305\u542b\u4e00\u4e2a\u6216\u591a\u4e2a\u5b50\u6a21\u578b\uff08\u5982chatglm2-6b)\u3002\u4e0b\u9762\u5bf9\u5404\u4e2a\u90e8\u5206\u8fdb\u884c\u5c55\u5f00\u4ecb\u7ecd\uff1a<\/p>\n

2. python\u6267\u884c\u73af\u5883<\/h2>\n

\u8be5\u90e8\u5206\u4e3a\u6a21\u578b\u63a8\u7406\u65f6\u9700\u8981\u7684\u76f8\u5173python\u4f9d\u8d56\u5305\uff0c\u53ef\u4ee5\u4f7f\u7528conda-pack\u5c06conda\u865a\u62df\u73af\u5883\u6253\u5305\uff0c\u5982python-3-8.tar.gz\u3002\u5982\u5bf9\u6253\u5305conda\u73af\u5883\u4e0d\u719f\u6089\u7684\uff0c\u53ef\u4ee5\u53c2\u8003 conda.github.io\/conda-pack\/<\/a>\u3002\u7136\u540e\u5728config.pbtxt\u4e2d\u914d\u7f6e\u6267\u884c\u73af\u5883\u8def\u5f84\uff1a<\/p>\n

<\/div>
css<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>parameters: {<\/span>\n  key: \"EXECUTION_ENV_PATH\"<\/span>,<\/span>\n  value: {string_value: \"$$TRITON_MODEL_DIRECTORY\/..\/python-3-8.tar.gz\"<\/span>}<\/span>\n}<\/span>\n<\/code><\/pre>\n
\u5728\u5f53\u524d\u793a\u4f8b\u4e2d\uff0c$$TRITON_MODEL_DIRECTORY=”$pwd\/model_repository\/chatglm2-6b”\u3002<\/p>\n
\u6ce8\u610f\uff1a\u5f53\u524dpython\u6267\u884c\u73af\u5883\u4e3a\u6240\u6709\u5b50\u6a21\u578b\u5171\u4eab\uff0c\u5982\u679c\u60f3\u7ed9\u4e0d\u540c\u5b50\u6a21\u578b\u6307\u5b9a\u4e0d\u540c\u7684\u6267\u884c\u73af\u5883\uff0c\u5219\u5e94\u8be5\u5c06tar.gz\u6587\u4ef6\u653e\u5728\u5b50\u6a21\u578b\u76ee\u5f55\u4e0b\uff0c\u5982\u4e0b\u6240\u793a\uff1a<\/p>\n
\n
\"chatglm2-6b\u6a21\u578b\u57289n-triton\u4e2d\u90e8\u7f72\u5e76\u96c6\u6210\u81f3langchain\u5b9e\u8df5<\/figure>\n<\/p>\n
\u540c\u65f6\uff0c\u5728config.pbtxt\u4e2d\u914d\u7f6e\u6267\u884c\u73af\u5883\u8def\u5f84\u5982\u4e0b\uff1a<\/p>\n
<\/div>
css<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>parameters: {<\/span>\n  key: \"EXECUTION_ENV_PATH\"<\/span>,<\/span>\n  value: {string_value: \"$$TRITON_MODEL_DIRECTORY\/python-3-8.tar.gz\"<\/span>}<\/span>\n}<\/span>\n<\/code><\/pre>\n
3. \u6a21\u578b\u914d\u7f6e\u6587\u4ef6<\/h2>\n
\u6a21\u578b\u4ed3\u5e93\u5e93\u4e2d\u7684\u6bcf\u4e2a\u6a21\u578b\u90fd\u5fc5\u987b\u5305\u542b\u4e00\u4e2a\u6a21\u578b\u914d\u7f6e\u6587\u4ef6config.pbtxt\uff0c\u7528\u4e8e\u6307\u5b9a\u5e73\u53f0\u548c\u6216\u540e\u7aef\u5c5e\u6027\u3001max_batch_size \u5c5e\u6027\u4ee5\u53ca\u6a21\u578b\u7684\u8f93\u5165\u548c\u8f93\u51fa\u5f20\u91cf\u7b49\u3002ChatGLM2-6B\u7684\u914d\u7f6e\u6587\u4ef6\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\uff1a<\/p>\n
<\/div>
yaml<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>name:<\/span> \"chatglm2-6b\"<\/span> \/\/<\/span> \u5fc5\u586b\uff0c\u6a21\u578b\u540d\uff0c\u9700\u4e0e\u8be5\u5b50\u6a21\u578b\u7684\u6587\u4ef6\u5939\u540d\u5b57\u76f8\u540c<\/span><\/span>\nbackend:<\/span> \"python\"<\/span> \/\/<\/span> \u5fc5\u586b\uff0c\u6a21\u578b\u6240\u4f7f\u7528\u7684\u540e\u7aef\u5f15\u64ce<\/span><\/span>\n<\/span>\nmax_batch_size:<\/span> 0<\/span> \/\/<\/span> \u6a21\u578b\u6bcf\u6b21\u8bf7\u6c42\u6700\u5927\u7684\u6279\u6570\u636e\u91cf\uff0c\u5f20\u91cfshape\u7531max_batch_size\u548cdims\u7ec4\u5408\u6307\u5b9a\uff0c\u5bf9\u4e8e<\/span> max_batch_size<\/span> \u5927\u4e8e<\/span> 0<\/span> \u7684\u6a21\u578b\uff0c\u5b8c\u6574\u5f62\u72b6\u5f62\u6210\u4e3a<\/span> [ -1<\/span> ] +<\/span> dims\u3002<\/span> \u5bf9\u4e8e<\/span> max_batch_size<\/span> \u7b49\u4e8e<\/span> 0<\/span> \u7684\u6a21\u578b\uff0c\u5b8c\u6574\u5f62\u72b6\u5f62\u6210\u4e3a<\/span> dims\u3002<\/span><\/span>\ninput<\/span> [ \/\/<\/span> \u5fc5\u586b\uff0c\u8f93\u5165\u5b9a\u4e49<\/span><\/span>\n  {<\/span>\n    name:<\/span> \"prompt\"<\/span> \/\/\u5fc5\u586b\uff0c\u540d\u79f0<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span> \/\/\u5fc5\u586b\uff0c\u6570\u636e\u7c7b\u578b<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ] \/\/\u5fc5\u586b\uff0c\u6570\u636e\u7ef4\u5ea6\uff0c-1<\/span> \u8868\u793a\u53ef\u53d8\u7ef4\u5ea6<\/span><\/span>\n  },<\/span>\n  {<\/span>\n    name:<\/span> \"history\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  },<\/span>\n  {<\/span>\n    name:<\/span> \"temperature\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  },<\/span>\n  {<\/span>\n    name:<\/span> \"max_token\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  },<\/span>\n  {<\/span>\n    name:<\/span> \"history_len\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  }<\/span>\n]<\/span>\noutput<\/span> [ \/\/\u5fc5\u586b\uff0c\u8f93\u51fa\u5b9a\u4e49<\/span><\/span>\n  {<\/span>\n    name:<\/span> \"response\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  },<\/span>\n  {<\/span>\n    name:<\/span> \"history\"<\/span><\/span>\n    data_type:<\/span> TYPE_STRING<\/span><\/span>\n    dims:<\/span> [ -1<\/span> ]<\/span>\n  }<\/span>\n]<\/span>\nparameters:<\/span> { \/\/\u6307\u5b9apython\u6267\u884c\u73af\u5883<\/span><\/span>\n  key:<\/span> \"EXECUTION_ENV_PATH\"<\/span>,<\/span>\n  value:<\/span> {string_value:<\/span> \"$$TRITON_MODEL_DIRECTORY\/..\/python-3-8.tar.gz\"<\/span>}<\/span>\n}<\/span>\ninstance_group<\/span> [ \/\/\u6a21\u578b\u5b9e\u4f8b\u7ec4<\/span><\/span>\n  { <\/span>\n      count:<\/span> 1<\/span>  \/\/\u5b9e\u4f8b\u6570\u91cf<\/span><\/span>\n      kind:<\/span> KIND_GPU<\/span>  \/\/\u5b9e\u4f8b\u7c7b\u578b<\/span><\/span>\n      gpus:<\/span> [ 0<\/span> ]  \/\/\u6307\u5b9a\u5b9e\u4f8b\u53ef\u7528\u7684GPU\u7d22\u5f15<\/span><\/span>\n  }<\/span>\n]<\/span>\n<\/code><\/pre>\n
\u5176\u4e2d\u5fc5\u586b\u9879\u4e3a\u6700\u5c0f\u6a21\u578b\u914d\u7f6e\uff0c\u6a21\u578b\u914d\u7f6e\u6587\u4ef6\u66f4\u591a\u4fe1\u606f\u53ef\u4ee5\u53c2\u8003\uff1a github.com\/triton-infe\u2026<\/a><\/p>\n
4. \u81ea\u5b9a\u4e49python backend<\/h2>\n
\u4e3b\u8981\u9700\u8981\u5b9e\u73b0model.py \u4e2d\u63d0\u4f9b\u7684\u4e09\u4e2a\u63a5\u53e3\uff1a<\/p>\n
\u2460. initialize: \u521d\u59cb\u5316\u8be5Python\u6a21\u578b\u65f6\u4f1a\u8fdb\u884c\u8c03\u7528\uff0c\u4e00\u822c\u6267\u884c\u83b7\u53d6\u8f93\u51fa\u4fe1\u606f\u53ca\u521b\u5efa\u6a21\u578b\u7684\u64cd\u4f5c<\/p>\n
\u2461. execute: python\u6a21\u578b\u63a5\u6536\u8bf7\u6c42\u65f6\u7684\u6267\u884c\u51fd\u6570\uff1b<\/p>\n
\u2462. finalize: \u5220\u9664\u6a21\u578b\u65f6\u4f1a\u8fdb\u884c\u8c03\u7528\uff1b<\/p>\n
\u5982\u679c\u6709 n \u4e2a\u6a21\u578b\u5b9e\u4f8b\uff0c\u90a3\u4e48\u4f1a\u8c03\u7528 n \u6b21initialize \u548c finalize\u8fd9\u4e24\u4e2a\u51fd\u6570\u3002<\/p>\n
ChatGLM2-6B\u7684model.py\u6587\u4ef6\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\uff1a<\/p>\n
<\/div>
python<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>import<\/span> os<\/span>\n# \u8bbe\u7f6e\u663e\u5b58\u7a7a\u95f2block\u6700\u5927\u5206\u5272\u9608\u503c<\/span><\/span>\nos.environ['PYTORCH_CUDA_ALLOC_CONF'<\/span>] = 'max_split_size_mb:32'<\/span><\/span>\n# \u8bbe\u7f6ework\u76ee\u5f55<\/span><\/span>\n<\/span>\nos.environ['TRANSFORMERS_CACHE'<\/span>] = os.path.dirname(os.path.abspath(__file__))+\"\/work\/\"<\/span><\/span>\nos.environ['HF_MODULES_CACHE'<\/span>] = os.path.dirname(os.path.abspath(__file__))+\"\/work\/\"<\/span><\/span>\n<\/span>\nimport<\/span> json<\/span>\n<\/span>\n# triton_python_backend_utils is available in every Triton Python model. You<\/span><\/span>\n# need to use this module to create inference requests and responses. It also<\/span><\/span>\n# contains some utility functions for extracting information from model_config<\/span><\/span>\n# and converting Triton input\/output types to numpy types.<\/span><\/span>\nimport<\/span> triton_python_backend_utils as<\/span> pb_utils<\/span>\nimport<\/span> sys<\/span>\nimport<\/span> gc<\/span>\nimport<\/span> time<\/span>\nimport<\/span> logging<\/span>\nimport<\/span> torch<\/span>\nfrom<\/span> transformers import<\/span> AutoTokenizer, AutoModel<\/span>\nimport<\/span> numpy as<\/span> np<\/span>\n<\/span>\ngc.collect()<\/span>\ntorch.cuda.empty_cache()<\/span>\n<\/span>\nlogging.basicConfig(format<\/span>='%(asctime)s - %(filename)s[line:%(lineno)d] - %(levelname)s: %(message)s'<\/span>,<\/span>\n                    level=logging.INFO)<\/span>\n<\/span>\nclass<\/span> TritonPythonModel<\/span>:<\/span>\n    \"\"\"Your Python model must use the same class name. Every Python model<\/span><\/span>\n    that is created must have \"TritonPythonModel\" as the class name.<\/span>\n    \"\"\"<\/span>\n<\/span>\n    def<\/span> initialize<\/span>(self, args<\/span>):<\/span>\n        \"\"\"`initialize` is called only once when the model is being loaded.<\/span><\/span>\n        Implementing `initialize` function is optional. This function allows<\/span>\n        the model to intialize any state associated with this model.<\/span>\n<\/span>\n        Parameters<\/span>\n        ----------<\/span>\n        args : dict<\/span>\n          Both keys and values are strings. The dictionary keys and values are:<\/span>\n          * model_config: A JSON string containing the model configuration<\/span>\n          * model_instance_kind: A string containing model instance kind<\/span>\n          * model_instance_device_id: A string containing model instance device ID<\/span>\n          * model_repository: Model repository path<\/span>\n          * model_version: Model version<\/span>\n          * model_name: Model name<\/span>\n        \"\"\"<\/span>\n        # You must parse model_config. JSON string is not parsed here<\/span><\/span>\n        self.model_config = json.loads(args['model_config'<\/span>])<\/span>\n        <\/span>\n        output_response_config = pb_utils.get_output_config_by_name(self.model_config, \"response\"<\/span>)<\/span>\n        output_history_config = pb_utils.get_output_config_by_name(self.model_config, \"history\"<\/span>)<\/span>\n<\/span>\n        # Convert Triton types to numpy types<\/span><\/span>\n        self.output_response_dtype = pb_utils.triton_string_to_numpy(output_response_config['data_type'<\/span>])<\/span>\n        self.output_history_dtype = pb_utils.triton_string_to_numpy(output_history_config['data_type'<\/span>])<\/span>\n        <\/span>\n        ChatGLM_path = os.path.dirname(os.path.abspath(__file__))+\"\/ChatGLM2_6B\"<\/span><\/span>\n        self.tokenizer = AutoTokenizer.from_pretrained(ChatGLM_path, trust_remote_code=True<\/span>)<\/span>\n        model = AutoModel.from_pretrained(ChatGLM_path,<\/span>\n                                          torch_dtype=torch.bfloat16,<\/span>\n                                          trust_remote_code=True<\/span>).half().cuda()<\/span>\n        self.model = model.eval<\/span>()<\/span>\n        logging.info(\"model init success\"<\/span>)<\/span>\n        <\/span>\n    def<\/span> execute<\/span>(self, requests<\/span>):<\/span>\n        \"\"\"`execute` MUST be implemented in every Python model. `execute`<\/span><\/span>\n        function receives a list of pb_utils.InferenceRequest as the only<\/span>\n        argument. This function is called when an inference request is made<\/span>\n        for this model. Depending on the batching configuration (e.g. Dynamic<\/span>\n        Batching) used, `requests` may contain multiple requests. Every<\/span>\n        Python model, must create one pb_utils.InferenceResponse for every<\/span>\n        pb_utils.InferenceRequest in `requests`. If there is an error, you can<\/span>\n        set the error argument when creating a pb_utils.InferenceResponse<\/span>\n<\/span>\n        Parameters<\/span>\n        ----------<\/span>\n        requests : list<\/span>\n          A list of pb_utils.InferenceRequest<\/span>\n<\/span>\n        Returns<\/span>\n        -------<\/span>\n        list<\/span>\n          A list of pb_utils.InferenceResponse. The length of this list must<\/span>\n          be the same as `requests`<\/span>\n          <\/span>\n        \"\"\"<\/span>\n        output_response_dtype = self.output_response_dtype<\/span>\n        output_history_dtype = self.output_history_dtype<\/span>\n<\/span>\n        # output_dtype = self.output_dtype<\/span><\/span>\n        responses = []<\/span>\n        # Every Python backend must iterate over everyone of the requests<\/span><\/span>\n        # and create a pb_utils.InferenceResponse for each of them.<\/span><\/span>\n        for<\/span> request in<\/span> requests:<\/span>\n            prompt = pb_utils.get_input_tensor_by_name(request, \"prompt\"<\/span>).as_numpy()[0<\/span>]<\/span>\n            prompt = prompt.decode('utf-8'<\/span>)<\/span>\n            history_origin = pb_utils.get_input_tensor_by_name(request, \"history\"<\/span>).as_numpy()<\/span>\n            if<\/span> len<\/span>(history_origin) > 0<\/span>:<\/span>\n                history = np.array([item.decode('utf-8'<\/span>) for<\/span> item in<\/span> history_origin]).reshape((-1<\/span>,2<\/span>)).tolist()<\/span>\n            else<\/span>:<\/span>\n                history = []<\/span>\n            temperature = pb_utils.get_input_tensor_by_name(request, \"temperature\"<\/span>).as_numpy()[0<\/span>]<\/span>\n            temperature = float<\/span>(temperature.decode('utf-8'<\/span>))<\/span>\n            max_token = pb_utils.get_input_tensor_by_name(request, \"max_token\"<\/span>).as_numpy()[0<\/span>]<\/span>\n            max_token = int<\/span>(max_token.decode('utf-8'<\/span>))<\/span>\n            history_len = pb_utils.get_input_tensor_by_name(request, \"history_len\"<\/span>).as_numpy()[0<\/span>]<\/span>\n            history_len = int<\/span>(history_len.decode('utf-8'<\/span>))<\/span>\n            <\/span>\n            # \u65e5\u5fd7\u8f93\u51fa\u4f20\u5165\u4fe1\u606f<\/span><\/span>\n            in_log_info = {<\/span>\n                \"in_prompt\"<\/span>:prompt,<\/span>\n                \"in_history\"<\/span>:history,<\/span>\n                \"in_temperature\"<\/span>:temperature,<\/span>\n                \"in_max_token\"<\/span>:max_token,<\/span>\n                \"in_history_len\"<\/span>:history_len<\/span>\n                       }<\/span>\n            logging.info(in_log_info)<\/span>\n            response,history = self.model.chat(self.tokenizer,<\/span>\n                                               prompt,<\/span>\n                                               history=history[-history_len:] if<\/span> history_len > 0<\/span> else<\/span> [],<\/span>\n                                               max_length=max_token,<\/span>\n                                               temperature=temperature)<\/span>\n            # \u65e5\u5fd7\u8f93\u51fa\u5904\u7406\u540e\u7684\u4fe1\u606f<\/span><\/span>\n            out_log_info = {<\/span>\n                \"out_response\"<\/span>:response,<\/span>\n                \"out_history\"<\/span>:history<\/span>\n                       }<\/span>\n            logging.info(out_log_info)<\/span>\n            response = np.array(response)<\/span>\n            history = np.array(history)<\/span>\n            <\/span>\n            response_output_tensor = pb_utils.Tensor(\"response\"<\/span>,response.astype(self.output_response_dtype))<\/span>\n            history_output_tensor = pb_utils.Tensor(\"history\"<\/span>,history.astype(self.output_history_dtype))<\/span>\n<\/span>\n            final_inference_response = pb_utils.InferenceResponse(output_tensors=[response_output_tensor,history_output_tensor])<\/span>\n            responses.append(final_inference_response)<\/span>\n            # Create InferenceResponse. You can set an error here in case<\/span><\/span>\n            # there was a problem with handling this inference request.<\/span><\/span>\n            # Below is an example of how you can set errors in inference<\/span><\/span>\n            # response:<\/span><\/span>\n            #<\/span><\/span>\n            # pb_utils.InferenceResponse(<\/span><\/span>\n            #    output_tensors=..., TritonError(\"An error occured\"))<\/span><\/span>\n<\/span>\n        # You should return a list of pb_utils.InferenceResponse. Length<\/span><\/span>\n        # of this list must match the length of `requests` list.<\/span><\/span>\n        return<\/span> responses<\/span>\n<\/span>\n    def<\/span> finalize<\/span>(self<\/span>):<\/span>\n        \"\"\"`finalize` is called only once when the model is being unloaded.<\/span><\/span>\n        Implementing `finalize` function is OPTIONAL. This function allows<\/span>\n        the model to perform any necessary clean ups before exit.<\/span>\n        \"\"\"<\/span>\n        print<\/span>('Cleaning up...'<\/span>)<\/span>\n<\/code><\/pre>\n
5. \u90e8\u7f72\u6d4b\u8bd5<\/h2>\n
\u2460 \u9009\u62e99n-triton-devel-gpu-v0.3\u955c\u50cf\u521b\u5efanotebook\u6d4b\u8bd5\u5b9e\u4f8b\uff1b<\/p>\n
\u2461 \u628a\u6a21\u578b\u653e\u5728\/9n-triton-devel\/model_repository\u76ee\u5f55\u4e0b\uff0c\u6a21\u578b\u76ee\u5f55\u7ed3\u6784\u53c2\u80033.1\uff1b<\/p>\n
\u2462 \u8fdb\u5165\/9n-triton-devel\/server\/\u76ee\u5f55\uff0c\u62c9\u53d6\u6700\u65b0\u7248\u672c\u7684bin\u5e76\u89e3\u538b\uff1awget storage.jd.local\/com.bamboo.\u2026<\/a><\/p>\n
\u2463 \u4fee\u6539\/9n-triton-devel\/server\/start.sh \u4e3a\u5982\u4e0b\uff1a<\/p>\n
<\/div>
bash<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>mkdir<\/span> logs<\/span>\nrm<\/span> -rf \/9n-triton-devel\/server\/logs\/*<\/span>\nrm<\/span> -rf \/tmp\/python_env_*<\/span>\nexport<\/span> LD_LIBRARY_PATH=\/9n-triton-devel\/server\/lib\/:$LD_LIBRARY_PATH<\/span><\/span>\nnohup<\/span> .\/bin\/9n_predictor_server --flagfile=.\/conf\/server.gflags 2>&1 >\/dev\/null &<\/span>\nsleep<\/span> 2<\/span>\npid=`ps x |grep \"9n_predictor_server\"<\/span> | grep -v \"grep\"<\/span> | grep -v \"ldd\"<\/span> | grep -v \"stat\"<\/span> | awk '{print $1}'<\/span>`<\/span>\necho<\/span> $pid<\/span><\/span>\n<\/code><\/pre>\n
\u2464 \u8fd0\u884c \/9n-triton-devel\/server\/start.sh \u811a\u672c<\/p>\n
\u2465 \u68c0\u67e5\u670d\u52a1\u542f\u52a8\u6210\u529f\uff08ChatGLM2-6B\u6a21\u578b\u542f\u52a8\uff0c\u5dee\u4e0d\u591a13\u5206\u949f\u5de6\u53f3\uff09<\/p>\n
\u65b9\u6cd51:\u67e5\u770b8010\u7aef\u53e3\u662f\u5426\u542f\u52a8\uff1anetstat -natp | grep 8010<\/p>\n
\u65b9\u6cd52:\u67e5\u770b\u65e5\u5fd7\uff1acat \/9n-triton-devel\/server\/logs\/predictor_core.INFO<\/p>\n
\u2466 \u7f16\u5199python grpc client\u8bbf\u95ee\u6d4b\u8bd5\u670d\u52a1\u811a\u672c\uff0c\u653e\u4e8e\/9n-triton-devel\/client\/\u76ee\u5f55\u4e0b\uff0c\u8bbf\u95ee\u7aef\u53e3\u4e3a8010\uff0cip\u4e3a127.0.0.1\uff0c\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\uff1a<\/p>\n
<\/div>
css<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>#!\/usr\/bin\/python3<\/span>\n# -*- coding: utf-8<\/span> -*-<\/span>\nimport sys<\/span>\nsys.path.append<\/span>('.\/base'<\/span>)<\/span>\nfrom multi_backend_client import MultiBackendClient<\/span>\nimport triton_python_backend_utils as python_backend_utils<\/span>\nimport multi_backend_message_pb2<\/span>\n<\/span>\nimport time<\/span>\nimport argparse<\/span>\nimport io<\/span>\nimport os<\/span>\nimport numpy as np<\/span>\nimport json<\/span>\nimport struct<\/span>\n<\/span>\ndef print_result<\/span>(response, batch_size ):<\/span>\n    print<\/span>(\"outputs len:\"<\/span> + str<\/span>(len<\/span>(response.outputs)))<\/span>\n<\/span>\n    if (response.error_code == 0<\/span>):<\/span>\n        print<\/span>(\"response : \"<\/span>, response)<\/span>\n<\/span>\n        print<\/span>(f'res shape: {response.outputs[0].shape}'<\/span>)<\/span>\n        res = python_backend_utils.deserialize_bytes_tensor<\/span>(response.raw_output_contents[0<\/span>])<\/span>\n        for i in res:<\/span>\n            print<\/span>(i.decode<\/span>())<\/span>\n<\/span>\n        print<\/span>(f'history shape: {response.outputs[1].shape}'<\/span>)<\/span>\n        history = python_backend_utils.deserialize_bytes_tensor<\/span>(response.raw_output_contents[1<\/span>])<\/span>\n        for i in history:<\/span>\n            print<\/span>(i.decode<\/span>())<\/span>\n<\/span>\n<\/span>\ndef send_one_request<\/span>(sender, request_pb, batch_size):<\/span>\n    succ, response = sender.send_req<\/span>(request_pb)<\/span>\n    if succ:<\/span>\n        print_result<\/span>(response, batch_size)<\/span>\n    else:<\/span>\n      print<\/span>('send_one_request fail '<\/span>, response)<\/span>\n<\/span>\ndef send_request<\/span>(ip, port, temperature, max_token, history_len, batch_size=1<\/span>, send_cnt=1<\/span>):<\/span>\n    request_sender = MultiBackendClient<\/span>(ip, port)<\/span>\n<\/span>\n    request = multi_backend_message_pb2.ModelInferRequest<\/span>()<\/span>\n    request.model_name = \"chatglm2-6b\"<\/span><\/span>\n<\/span>\n    # \u8f93\u5165\u5360\u4f4d<\/span>\n    input0 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferInputTensor<\/span>()<\/span>\n    input0.name = \"prompt\"<\/span><\/span>\n    input0.datatype = \"BYTES\"<\/span><\/span>\n    input0.shape.extend<\/span>([1<\/span>])<\/span>\n<\/span>\n    input1 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferInputTensor<\/span>()<\/span>\n    input1.name = \"history\"<\/span><\/span>\n    input1.datatype = \"BYTES\"<\/span><\/span>\n    input1.shape.extend<\/span>([-1<\/span>])<\/span>\n    <\/span>\n    input2 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferInputTensor<\/span>()<\/span>\n    input2.name = \"temperature\"<\/span><\/span>\n    input2.datatype = \"BYTES\"<\/span><\/span>\n    input2.shape.extend<\/span>([1<\/span>])<\/span>\n    <\/span>\n    input3 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferInputTensor<\/span>()<\/span>\n    input3.name = \"max_token\"<\/span><\/span>\n    input3.datatype = \"BYTES\"<\/span><\/span>\n    input3.shape.extend<\/span>([1<\/span>])<\/span>\n    <\/span>\n    input4 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferInputTensor<\/span>()<\/span>\n    input4.name = \"history_len\"<\/span><\/span>\n    input4.datatype = \"BYTES\"<\/span><\/span>\n    input4.shape.extend<\/span>([1<\/span>])<\/span>\n<\/span>\n    query = '\u8bf7\u7ed9\u51fa\u4e00\u4e2a\u5177\u4f53\u793a\u4f8b'<\/span><\/span>\n    input0.contents.bytes_contents.append<\/span>(bytes<\/span>(query, encoding=\"utf8\"<\/span>))<\/span>\n    request.inputs.extend<\/span>([input0])<\/span>\n<\/span>\n    history_origin = np.array<\/span>([['\u4f60\u77e5\u9053\u9e21\u5154\u540c\u7b3c\u95ee\u9898\u4e48'<\/span>, '\u9e21\u5154\u540c\u7b3c\u95ee\u9898\u662f\u4e00\u4e2a\u7ecf\u5178\u7684\u6570\u5b66\u95ee\u9898\uff0c\u6d89\u53ca\u5230\u57fa\u672c\u7684\u4ee3\u6570\u65b9\u7a0b\u548c\u89e3\u9898\u65b9\u6cd5\u3002\u95ee\u9898\u63cf\u8ff0\u4e3a\uff1a\u5728\u4e00\u4e2a\u7b3c\u5b50\u91cc\u9762\uff0c\u6709\u82e5\u5e72\u53ea\u9e21\u548c\u5154\u5b50\uff0c\u5df2\u77e5\u5b83\u4eec\u7684\u603b\u6570\u548c\u603b\u817f\u6570\uff0c\u95ee\u9e21\u548c\u5154\u5b50\u7684\u6570\u91cf\u5404\u662f\u591a\u5c11\uff1fnn\u89e3\u6cd5\u5982\u4e0b\uff1a\u5047\u8bbe\u9e21\u7684\u6570\u91cf\u4e3ax\uff0c\u5154\u5b50\u7684\u6570\u91cf\u4e3ay\uff0c\u5219\u603b\u817f\u6570\u4e3a2x+4y\u3002\u6839\u636e\u9898\u610f\uff0c\u53ef\u4ee5\u5217\u51fa\u65b9\u7a0b\u7ec4\uff1annx + y = \u603b\u6570n2x + 4y = \u603b\u817f\u6570nn\u901a\u8fc7\u89e3\u65b9\u7a0b\u7ec4\uff0c\u53ef\u4ee5\u6c42\u5f97x\u548cy\u7684\u503c\uff0c\u4ece\u800c\u786e\u5b9a\u9e21\u548c\u5154\u5b50\u7684\u6570\u91cf\u3002'<\/span>]]).reshape<\/span>((-1<\/span>,))<\/span>\n    history = [bytes<\/span>(item, encoding=\"utf8\"<\/span>) for item in history_origin]<\/span>\n    input1.contents.bytes_contents.extend<\/span>(history)<\/span>\n    request.inputs.extend<\/span>([input1])<\/span>\n    <\/span>\n    input2.contents.bytes_contents.append<\/span>(bytes<\/span>(temperature, encoding=\"utf8\"<\/span>))<\/span>\n    request.inputs.extend<\/span>([input2])<\/span>\n    <\/span>\n    input3.contents.bytes_contents.append<\/span>(bytes<\/span>(max_token, encoding=\"utf8\"<\/span>))<\/span>\n    request.inputs.extend<\/span>([input3])<\/span>\n    <\/span>\n    input4.contents.bytes_contents.append<\/span>(bytes<\/span>(history_len, encoding=\"utf8\"<\/span>))<\/span>\n    request.inputs.extend<\/span>([input4])<\/span>\n<\/span>\n    # \u8f93\u51fa\u5360\u4f4d<\/span>\n    output_tensor0 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferRequestedOutputTensor<\/span>()<\/span>\n    output_tensor0.name = \"response\"<\/span><\/span>\n    request.outputs.extend<\/span>([output_tensor0])<\/span>\n<\/span>\n    output_tensor1 = multi_backend_message_pb2.ModelInferRequest<\/span>().InferRequestedOutputTensor<\/span>()<\/span>\n    output_tensor1.name = \"history\"<\/span><\/span>\n    request.outputs.extend<\/span>([output_tensor1])<\/span>\n    <\/span>\n    min_ms = 0<\/span><\/span>\n    max_ms = 0<\/span><\/span>\n    avg_ms = 0<\/span><\/span>\n    for i in range<\/span>(send_cnt):<\/span>\n        start = time.time_ns<\/span>()<\/span>\n        send_one_request<\/span>(request_sender, request, batch_size)<\/span>\n        cost = (time.time_ns<\/span>()-start)\/1000000<\/span><\/span>\n        print (\"idx:%d cost  ms:%d\"<\/span> % (i, cost))<\/span>\n        if cost > max_ms:<\/span>\n            max_ms = cost<\/span>\n        if cost < min_ms or min_ms==0<\/span>:<\/span>\n            min_ms = cost<\/span>\n        avg_ms += cost<\/span>\n    avg_ms \/= send_cnt<\/span>\n    print<\/span>(\"cnt=%d max=%dms min=%dms avg=%dms\"<\/span> % (send_cnt, max_ms, min_ms, avg_ms))<\/span>\n<\/span>\nif __name__ == '__main__'<\/span>:<\/span>\n        parser = argparse.ArgumentParser<\/span>()<\/span>\n        parser.add_argument<\/span>( '-ip'<\/span>, '--ip_address'<\/span>, help = 'ip address'<\/span>, default='127.0.0.1'<\/span>, required=False)<\/span>\n        parser.add_argument<\/span>( '-p'<\/span>,  '--port'<\/span>, help = 'port'<\/span>, default='8010'<\/span>, required=False)<\/span>\n        parser.add_argument<\/span>( '-t'<\/span>,  '--temperature'<\/span>, help = 'temperature'<\/span>, default='0.01'<\/span>, required=False)<\/span>\n        parser.add_argument<\/span>( '-m'<\/span>,  '--max_token'<\/span>, help = 'max_token'<\/span>, default='16000'<\/span>, required=False)<\/span>\n        parser.add_argument<\/span>( '-hl'<\/span>,  '--history_len'<\/span>, help = 'history_len'<\/span>, default='10'<\/span>, required=False)<\/span>\n        parser.add_argument<\/span>( '-b'<\/span>,  '--batch_size'<\/span>, help = 'batch size'<\/span>, default=1<\/span>, required=False, type = int)<\/span>\n        parser.add_argument<\/span>( '-c'<\/span>,  '--send_count'<\/span>, help = 'send count'<\/span>, default=1<\/span>, required=False, type = int)<\/span>\n        args = parser.parse_args<\/span>()<\/span>\n        send_request<\/span>(args.ip_address, args.port, args.temperature, args.max_token, args.history_len, args.batch_size, args.send_count)<\/span>\n<\/code><\/pre>\n
6. \u6a21\u578b\u90e8\u7f72<\/h2>\n
\u4e5d\u6570\u7b97\u6cd5\u4e2d\u53f0\u63d0\u4f9b\u4e86\u4e24\u79cd\u90e8\u7f72\u6a21\u578b\u670d\u52a1\u65b9\u5f0f\uff0c\u5206\u522b\u4e3a\u754c\u9762\u90e8\u7f72\u548cSDK\u90e8\u7f72\u3002\u5229\u7528\u754c\u9762\u4e2d\u7684\u6a21\u578b\u90e8\u7f72\u53ea\u652f\u6301JSF\u534f\u8bae\u63a5\u53e3\uff0c\u82e5\u8981\u63d0\u4f9bJSF\u670d\u52a1\u63a5\u53e3\uff0c\u5219\u53ef\u4ee5\u53c2\u8003 easyalgo.jd.com\/help\/%E4%BD\u2026<\/a> \u76f4\u63a5\u90e8\u7f72\u3002<\/p>\n
\u7531\u4e8e\u6211\u540e\u7eed\u9700\u8981\u5c06ChatGLM2-6B\u6a21\u578b\u96c6\u6210\u81f3langchain\u4e2d\u4f7f\u7528\uff0c\u6240\u4ee5\u5bf9\u5916\u63d0\u4f9bhttp\u534f\u8bae\u63a5\u53e3\u6bd4\u8f83\u4fbf\u5229\uff0c\u7ecf\u4e0e\u7b97\u6cd5\u4e2d\u53f0\u540c\u5b66\u8bf7\u6559\u540e\u4f7f\u7528SDK\u65b9\u5f0f\u90e8\u7f72\u53ef\u4ee5\u6ee1\u8db3\u3002\u7531\u4e8e\u754c\u9762\u90e8\u7f72\u548cSDK\u90e8\u7f72\u76ee\u524d\u7814\u53d1\u6ca1\u6709\u5bf9\u9f50\uff0c\u7528\u754c\u9762\u90e8\u7f72\u65f6\u76f4\u63a5\u53ef\u4ee5\u4f7f\u75283.1\u4e2d\u7684\u6a21\u578b\u7ed3\u6784\uff0c\u4f7f\u7528SDK\u90e8\u7f72\u5219\u9700\u8981\u8c03\u6574\u6a21\u578b\u7ed3\u6784\u5982\u4e0b\uff1a<\/p>\n
\n
\"chatglm2-6b\u6a21\u578b\u57289n-triton\u4e2d\u90e8\u7f72\u5e76\u96c6\u6210\u81f3langchain\u5b9e\u8df5<\/figure>\n<\/p>\n
\u540c\u65f6\u9700\u8981\u5728config.pbtxt\u4e2d\u5c06\u6267\u884c\u73af\u5883\u8def\u5f84\u8bbe\u7f6e\u5982\u4e0b\uff1a<\/p>\n
<\/div>
css<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>parameters: {<\/span>\n  key: \"EXECUTION_ENV_PATH\"<\/span>,<\/span>\n  value: {string_value: \"$$TRITON_MODEL_DIRECTORY\/1\/python-3-8.tar.gz\"<\/span>}<\/span>\n}<\/span>\n<\/code><\/pre>\n
\u6a21\u578b\u90e8\u7f72\u4ee3\u7801\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\uff1a<\/p>\n
<\/div>
ini<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div>from das.triton.model import TritonModel<\/span>\n<\/span>\nmodel<\/span> = TritonModel(\"chatglm2-6b\"<\/span>)<\/span>\n<\/span>\npredictor<\/span> = model.deploy(<\/span>\n    path<\/span>=\"$pwd\/model_repository\/chatglm2-6b\"<\/span>, # \u6a21\u578b\u6587\u4ef6\u6240\u5728\u7684\u76ee\u5f55<\/span><\/span>\n    protocol<\/span>='http'<\/span>,<\/span>\n    endpoint<\/span> = \"9n-das-serving-lf2.jd.local\"<\/span>,<\/span>\n    cpu<\/span>=4<\/span>,<\/span>\n    memory<\/span>=30<\/span>,<\/span>\n    use_gpu<\/span>=True<\/span>, # \u6839\u636e\u662f\u5426\u9700\u8981gpu\u52a0\u901f\u63a8\u7406\u6765\u914d\u7f6e<\/span><\/span>\n    override<\/span> = True<\/span>,<\/span>\n    instances<\/span>=2<\/span><\/span>\n    )<\/span>\n<\/code><\/pre>\n
\u56db.\u96c6\u6210\u81f3langchain<\/h1>\n
\u4f7f\u7528langchain\u53ef\u4ee5\u5feb\u901f\u57fa\u4e8eLLM\u6a21\u578b\u5f00\u53d1\u4e00\u4e9b\u5e94\u7528\u3002\u4f7f\u7528LLMs\u6a21\u5757\u5c01\u88c5ChatGLM2-6B\uff0c\u8bf7\u6c42\u6211\u4eec\u7684\u6a21\u578b\u670d\u52a1\uff0c\u4e3b\u8981\u5b9e\u73b0_call\u51fd\u6570\uff0c\u53ef\u4ee5\u53c2\u8003\u5982\u4e0b\u4ee3\u7801\uff1a<\/p>\n
<\/div>
python<\/span>
\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><\/span>\nimport<\/span> json<\/span>\nimport<\/span> time<\/span>\nimport<\/span> base64<\/span>\nimport<\/span> struct<\/span>\nimport<\/span> requests<\/span>\nimport<\/span> numpy as<\/span> np<\/span>\nfrom<\/span> pathlib import<\/span> Path<\/span>\nfrom<\/span> abc import<\/span> ABC, abstractmethod<\/span>\nfrom<\/span> langchain.llms.base import<\/span> LLM<\/span>\nfrom<\/span> langchain.llms import<\/span> OpenAI<\/span>\nfrom<\/span> langchain.llms.utils import<\/span> enforce_stop_tokens<\/span>\nfrom<\/span> typing import<\/span> Dict<\/span>, List<\/span>, Optional<\/span>, Tuple<\/span>, Union<\/span>, Mapping, Any<\/span><\/span>\n<\/span>\nimport<\/span> warnings<\/span>\nwarnings.filterwarnings(\"ignore\"<\/span>)<\/span>\n<\/span>\nclass<\/span> ChatGLM<\/span>(LLM<\/span>):<\/span>\n    max_token = 32000<\/span><\/span>\n    temperature = 0.01<\/span><\/span>\n    history_len = 10<\/span><\/span>\n    url = \"\"<\/span><\/span>\n    def<\/span> __init__<\/span>(self<\/span>):<\/span>\n        super<\/span>(ChatGLM, self).__init__()<\/span>\n        <\/span>\n    @property<\/span><\/span>\n    def<\/span> _llm_type<\/span>(self<\/span>):<\/span>\n        return<\/span> \"ChatGLM2-6B\"<\/span><\/span>\n    <\/span>\n    @property<\/span><\/span>\n    def<\/span> _history_len<\/span>(self<\/span>) -> int<\/span>:<\/span>\n        return<\/span> self.history_len<\/span>\n    <\/span>\n    @property<\/span><\/span>\n    def<\/span> _max_token<\/span>(self<\/span>) -> int<\/span>:<\/span>\n        return<\/span> self.max_token<\/span>\n    <\/span>\n    @property<\/span><\/span>\n    def<\/span> _temperature<\/span>(self<\/span>) -> float<\/span>:<\/span>\n        return<\/span> self.temperature<\/span>\n    <\/span>\n    def<\/span> _deserialize_bytes_tensor<\/span>(self, encoded_tensor<\/span>):<\/span>\n        \"\"\"<\/span><\/span>\n        Deserializes an encoded bytes tensor into an<\/span>\n        numpy array of dtype of python objects<\/span>\n        Parameters<\/span>\n        ----------<\/span>\n        encoded_tensor : bytes<\/span>\n            The encoded bytes tensor where each element<\/span>\n            has its length in first 4 bytes followed by<\/span>\n            the content<\/span>\n        Returns<\/span>\n        -------<\/span>\n        string_tensor : np.array<\/span>\n            The 1-D numpy array of type object containing the<\/span>\n            deserialized bytes in 'C' order.<\/span>\n        \"\"\"<\/span>\n        strs = list<\/span>()<\/span>\n        offset = 0<\/span><\/span>\n        val_buf = encoded_tensor<\/span>\n        while<\/span> offset < len<\/span>(val_buf):<\/span>\n            l = struct.unpack_from(\"<I\"<\/span>, val_buf, offset)[0<\/span>]<\/span>\n            offset += 4<\/span><\/span>\n            sb = struct.unpack_from(\"<{}s\"<\/span>.format<\/span>(l), val_buf, offset)[0<\/span>]<\/span>\n            offset += l<\/span>\n            strs.append(sb)<\/span>\n        return<\/span> (np.array(strs, dtype=np.object_))<\/span>\n    <\/span>\n    @classmethod<\/span><\/span>\n    def<\/span> _infer<\/span>(cls, url, query, history, temperature, max_token, history_len<\/span>):<\/span>\n        query = base64.b64encode(query.encode('utf-8'<\/span>)).decode('utf-8'<\/span>)<\/span>\n        history_origin = np.asarray(history).reshape((-1<\/span>,))<\/span>\n        history = [base64.b64encode(item.encode('utf-8'<\/span>)).decode('utf-8'<\/span>) for<\/span> item in<\/span> history_origin]<\/span>\n        temperature = base64.b64encode(temperature.encode('utf-8'<\/span>)).decode('utf-8'<\/span>)<\/span>\n        max_token = base64.b64encode(max_token.encode('utf-8'<\/span>)).decode('utf-8'<\/span>)<\/span>\n        history_len = base64.b64encode(history_len.encode('utf-8'<\/span>)).decode('utf-8'<\/span>)<\/span>\n        data = {<\/span>\n            \"model_name\"<\/span>: \"chatglm2-6b\"<\/span>,<\/span>\n            \"inputs\"<\/span>: [<\/span>\n                {\"name\"<\/span>: \"prompt\"<\/span>, \"datatype\"<\/span>: \"BYTES\"<\/span>, \"shape\"<\/span>: [1<\/span>], \"contents\"<\/span>: {\"bytes_contents\"<\/span>: [query]}},<\/span>\n                {\"name\"<\/span>: \"history\"<\/span>, \"datatype\"<\/span>: \"BYTES\"<\/span>, \"shape\"<\/span>: [-1<\/span>], \"contents\"<\/span>: {\"bytes_contents\"<\/span>: history}},<\/span>\n                {\"name\"<\/span>: \"temperature\"<\/span>, \"datatype\"<\/span>: \"BYTES\"<\/span>, \"shape\"<\/span>: [1<\/span>], \"contents\"<\/span>: {\"bytes_contents\"<\/span>: [temperature]}},<\/span>\n                {\"name\"<\/span>: \"max_token\"<\/span>, \"datatype\"<\/span>: \"BYTES\"<\/span>, \"shape\"<\/span>: [1<\/span>], \"contents\"<\/span>: {\"bytes_contents\"<\/span>: [max_token]}},<\/span>\n                {\"name\"<\/span>: \"history_len\"<\/span>, \"datatype\"<\/span>: \"BYTES\"<\/span>, \"shape\"<\/span>: [1<\/span>], \"contents\"<\/span>: {\"bytes_contents\"<\/span>: [history_len]}}<\/span>\n                ],<\/span>\n            \"outputs\"<\/span>: [{\"name\"<\/span>: \"response\"<\/span>},<\/span>\n                        {\"name\"<\/span>: \"history\"<\/span>}]<\/span>\n            }<\/span>\n        response = requests.post(url = url, <\/span>\n                                 data = json.dumps(data, ensure_ascii=True<\/span>), <\/span>\n                                 headers = {\"Content_Type\"<\/span>: \"application\/json\"<\/span>}, <\/span>\n                                 timeout=120<\/span>)<\/span>\n        return<\/span> response <\/span>\n    <\/span>\n    def<\/span> _call<\/span>(self, <\/span><\/span>\n              query: str<\/span>, <\/span>\n              history: List<\/span>[List<\/span>[str<\/span>]] =[], <\/span>\n              stop: Optional<\/span>[List<\/span>[str<\/span>]] =None<\/span>):<\/span>\n        temperature = str<\/span>(self.temperature)<\/span>\n        max_token = str<\/span>(self.max_token)<\/span>\n        history_len = str<\/span>(self.history_len)<\/span>\n        url = self.url<\/span>\n        response = self._infer(url, query, history, temperature, max_token, history_len)<\/span>\n        if<\/span> response.status_code!=200<\/span>:<\/span>\n            return<\/span> \"\u67e5\u8be2\u7ed3\u679c\u9519\u8bef\"<\/span><\/span>\n        if<\/span> stop is<\/span> not<\/span> None<\/span>:<\/span>\n            response = enforce_stop_tokens(response, stop)<\/span>\n        result = json.loads(response.text)<\/span>\n        # \u5904\u7406response<\/span><\/span>\n        res = base64.b64decode(result['raw_output_contents'<\/span>][0<\/span>].encode('utf-8'<\/span>))<\/span>\n        res_response = self._deserialize_bytes_tensor(res)[0<\/span>].decode()<\/span>\n        return<\/span> res_response<\/span>\n    <\/span>\n    def<\/span> chat<\/span>(self, <\/span><\/span>\n              query: str<\/span>, <\/span>\n              history: List<\/span>[List<\/span>[str<\/span>]] =[], <\/span>\n              stop: Optional<\/span>[List<\/span>[str<\/span>]] =None<\/span>):<\/span>\n        temperature = str<\/span>(self.temperature)<\/span>\n        max_token = str<\/span>(self.max_token)<\/span>\n        history_len = str<\/span>(self.history_len)<\/span>\n        url = self.url<\/span>\n        response = self._infer(url, query, history, temperature, max_token, history_len)<\/span>\n        if<\/span> response.status_code!=200<\/span>:<\/span>\n            return<\/span> \"\u67e5\u8be2\u7ed3\u679c\u9519\u8bef\"<\/span><\/span>\n        if<\/span> stop is<\/span> not<\/span> None<\/span>:<\/span>\n            response = enforce_stop_tokens(response, stop)<\/span>\n        result = json.loads(response.text)<\/span>\n        # \u5904\u7406response<\/span><\/span>\n        res = base64.b64decode(result['raw_output_contents'<\/span>][0<\/span>].encode('utf-8'<\/span>))<\/span>\n        res_response = self._deserialize_bytes_tensor(res)[0<\/span>].decode()<\/span>\n        # \u5904\u7406history<\/span><\/span>\n        history_shape = result['outputs'<\/span>][1<\/span>][\"shape\"<\/span>]<\/span>\n        history_enc = base64.b64decode(result['raw_output_contents'<\/span>][1<\/span>].encode('utf-8'<\/span>))<\/span>\n        res_history = np.array([i.decode() for<\/span> i in<\/span> self._deserialize_bytes_tensor(history_enc)]).reshape(history_shape).tolist()<\/span>\n        return<\/span> res_response, res_history<\/span>\n    <\/span>\n    @property<\/span><\/span>\n    def<\/span> _identifying_params<\/span>(self<\/span>) -> Mapping[str<\/span>, Any<\/span>]:<\/span>\n        \"\"\"Get the identifying parameters.<\/span><\/span>\n        \"\"\"<\/span>\n        _param_dict = {<\/span>\n            \"url\"<\/span>: self.url<\/span>\n        }<\/span>\n        return<\/span> _param_dict<\/span>\n<\/code><\/pre>\n
\u6ce8\u610f\uff1a\u6a21\u578b\u670d\u52a1\u8c03\u7528url\u7b49\u4e8e\u5728\u6a21\u578b\u90e8\u7f72\u9875\u9762\u8c03\u7528\u4fe1\u606fURL\u540e\u52a0\u4e0a” MutilBackendService\/Predict<\/a> “<\/p>\n
\u4e94.\u603b\u7ed3<\/h1>\n
\u672c\u6587\u8be6\u7ec6\u4ecb\u7ecd\u4e86\u5728\u96c6\u56e29n-triton\u5de5\u5177\u4e0a\u90e8\u7f72ChatGLM2-6B\u8fc7\u7a0b\uff0c\u5e0c\u671b\u53ef\u4ee5\u4e3a\u6709\u90e8\u7f72\u9700\u6c42\u7684\u540c\u5b66\u63d0\u4f9b\u4e00\u4e9b\u5e2e\u52a9\u3002<\/p>\n
\n
\u4f5c\u8005\uff1a\u4eac\u4e1c\u4fdd\u9669 \u8d75\u98ce\u9f99<\/p>\n
\u6765\u6e90\uff1a\u4eac\u4e1c\u4e91\u5f00\u53d1\u8005\u793e\u533a \u8f6c\u8f7d\u8bf7\u6ce8\u660e\u51fa\u5904<\/p>\n<\/blockquote>","protected":false},"excerpt":{"rendered":"
\u672c\u6587\u5c06\u4ecb\u7ecd\u6211\u5229\u7528\u96c6\u56e29n-triton\u5de5\u5177\u90e8\u7f72ChatGLM2-6B\u8fc7\u7a0b\u4e2d\u8e29\u8fc7\u7684\u4e00\u4e9b\u5751\uff0c\u5e0c\u671b\u53ef\u4ee5\u4e3a\u6709\u90e8\u7f72\u9700\u6c42\u7684\u540c\u5b66\u63d0\u4f9b\u4e00\u4e9b\u5e2e\u52a9\u3002<\/p>\n","protected":false},"author":1,"featured_media":4419,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"rank_math_title":"ChatGLM-6B \u90e8\u7f72\u4e0e\u5e94\u7528\uff1aTriton & Langchain \u5b9e\u6218 - \u5948\u65af\u79d1\u6280\u793e\u533a","rank_math_description":"\u672c\u6587\u8be6\u7ec6\u4ecb\u7ecd\u4e86\u5982\u4f55\u4f7f\u7528 Triton \u90e8\u7f72 ChatGLM-6B \u6a21\u578b\uff0c\u5e76\u96c6\u6210\u81f3 langchain \u8fdb\u884c\u5e94\u7528\u5f00\u53d1\u3002\u6db5\u76d6\u4e86\u73af\u5883\u914d\u7f6e\u3001\u6a21\u578b\u8f6c\u6362\u3001Python Backend \u7f16\u5199\u3001\u90e8\u7f72\u6d4b\u8bd5\u53ca\u96c6\u6210\u793a\u4f8b\uff0c\u4e3a\u5f00\u53d1\u8005\u63d0\u4f9b\u5b9e\u7528\u6307\u5357\u3002","rank_math_focus_keyword":"ChatGLM, Triton \u90e8\u7f72, LLM, langchain, \u6a21\u578b\u63a8\u7406","views":"11","footnotes":""},"categories":[3],"tags":[126,127,128,129,136],"collection":[],"class_list":["post-1277","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fenlei2","tag-gpt","tag-ai","tag-128","tag-129","tag-136"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts\/1277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/comments?post=1277"}],"version-history":[{"count":0,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts\/1277\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/media\/4419"}],"wp:attachment":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/media?parent=1277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/categories?post=1277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/tags?post=1277"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/collection?post=1277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\u901a\u7528predictor\u8bf7\u6c42\u683c\u5f0f\u53ef\u4ee5\u53c2\u8003\uff1a github.com\/kserve\/kser\u2026<\/a><\/p>\n