{"id":1359,"date":"2024-05-06T07:31:13","date_gmt":"2024-05-06T07:31:13","guid":{"rendered":"https:\/\/www.nicekj.com\/?p=1359"},"modified":"2024-05-06T07:31:24","modified_gmt":"2024-05-06T07:31:24","slug":"langchainxiliejiaochengshiyongzhishikuxiufuhuanjue","status":"publish","type":"post","link":"https:\/\/www.nicekj.com\/langchainxiliejiaochengshiyongzhishikuxiufuhuanjue.html","title":{"rendered":"LangChain\u7cfb\u5217\u6559\u7a0b\uff1a\u4f7f\u7528\u77e5\u8bc6\u5e93\u4fee\u590d\u5e7b\u89c9"},"content":{"rendered":"<p>\u5927\u578b\u8bed\u8a00\u6a21\u578b\uff08LLMs\uff09\u5b58\u5728\u6570\u636e\u65b0\u9c9c\u5ea6\u95ee\u9898\u3002\u5373\u4f7f\u662f\u50cfGPT-4\u8fd9\u6837\u6700\u5f3a\u5927\u7684\u6a21\u578b\uff0c\u4e5f\u4e0d\u4e86\u89e3\u6700\u8fd1\u7684\u4e8b\u4ef6\u3002<\/p>\n<p>\u6839\u636eLLMs\u7684\u89c6\u89d2\uff0c\u4e16\u754c\u4eff\u4f5b\u505c\u6ede\u5728\u67d0\u4e2a\u65f6\u95f4\u70b9\u3002\u5b83\u4eec\u53ea\u77e5\u9053\u4e16\u754c\u662f\u5982\u4f55\u5728\u5b83\u4eec\u7684\u8bad\u7ec3\u6570\u636e\u4e2d\u5448\u73b0\u7684\u3002<\/p>\n<p>\u8fd9\u5bf9\u4e8e\u4f9d\u8d56\u6700\u65b0\u4fe1\u606f\u6216\u7279\u5b9a\u6570\u636e\u96c6\u7684\u4efb\u4f55\u7528\u4f8b\u90fd\u4f1a\u5e26\u6765\u95ee\u9898\u3002\u4f8b\u5982\uff0c\u60a8\u53ef\u80fd\u6709\u4e00\u4e9b\u5185\u90e8\u516c\u53f8\u6587\u4ef6\uff0c\u60a8\u5e0c\u671b\u901a\u8fc7LLM\u4e0e\u4e4b\u4e92\u52a8\u3002<\/p>\n<p>\u7b2c\u4e00\u4e2a\u6311\u6218\u662f\u5c06\u8fd9\u4e9b\u6587\u4ef6\u6dfb\u52a0\u5230LLM\u4e2d\uff0c\u6211\u4eec\u53ef\u4ee5\u5c1d\u8bd5\u8bad\u7ec3LLM\u4f7f\u7528\u8fd9\u4e9b\u6587\u4ef6\uff0c\u4f46\u8fd9\u662f\u8017\u65f6\u4e14\u6602\u8d35\u7684\u3002\u800c\u4e14\u5f53\u6dfb\u52a0\u65b0\u6587\u4ef6\u65f6\u4f1a\u53d1\u751f\u4ec0\u4e48\u5462\uff1f\u4e3a\u6bcf\u4e2a\u65b0\u6587\u4ef6\u8fdb\u884c\u8bad\u7ec3\u662f\u975e\u5e38\u4f4e\u6548\u7684\uff0c\u7b80\u76f4\u662f\u4e0d\u53ef\u80fd\u7684\u3002<\/p>\n<p>\u90a3\u4e48\uff0c\u6211\u4eec\u5982\u4f55\u5904\u7406\u8fd9\u4e2a\u95ee\u9898\u5462\uff1f\u6211\u4eec\u53ef\u4ee5\u4f7f\u7528\u68c0\u7d22\u589e\u5f3a\u6280\u672f\u3002\u8fd9\u79cd\u6280\u672f\u5141\u8bb8\u6211\u4eec\u4ece\u5916\u90e8\u77e5\u8bc6\u5e93\u4e2d\u68c0\u7d22\u76f8\u5173\u4fe1\u606f\uff0c\u5e76\u5c06\u8fd9\u4e9b\u4fe1\u606f\u63d0\u4f9b\u7ed9\u6211\u4eec\u7684LLM\u3002<\/p>\n<p>\u5916\u90e8\u77e5\u8bc6\u5e93\u5c31\u662f\u6211\u4eec\u4e86\u89e3LLM\u8bad\u7ec3\u6570\u636e\u4e4b\u5916\u4e16\u754c\u7684&#8221;\u7a97\u53e3&#8221;\u3002\u5728\u672c\u7ae0\u4e2d\uff0c\u6211\u4eec\u5c06\u5b66\u4e60\u5982\u4f55\u4f7f\u7528LangChain\u4e3aLLMs\u5b9e\u65bd\u68c0\u7d22\u589e\u5f3a\u3002<\/p>\n<h1 data-id=\"heading-0\">\u521b\u5efa\u77e5\u8bc6\u5e93<\/h1>\n<p>\u6211\u4eec\u6709\u4e24\u79cd\u4e3b\u8981\u7c7b\u578b\u7684\u77e5\u8bc6\u9002\u7528\u4e8eLLMs\u3002\u53c2\u6570\u5316\u77e5\u8bc6\u6307\u7684\u662fLLM\u5728\u8bad\u7ec3\u8fc7\u7a0b\u4e2d\u5b66\u5230\u7684\u4e00\u5207\uff0c\u5b83\u5145\u5f53\u4e86LLM\u7684\u4e16\u754c\u7684\u51bb\u7ed3\u5feb\u7167\u3002<\/p>\n<p>\u7b2c\u4e8c\u79cd\u7c7b\u578b\u7684\u77e5\u8bc6\u662f\u6e90\u77e5\u8bc6\u3002\u8fd9\u79cd\u77e5\u8bc6\u5305\u62ec\u901a\u8fc7\u8f93\u5165\u63d0\u793a\u8f93\u5165\u5230LLM\u4e2d\u7684\u4efb\u4f55\u4fe1\u606f\u3002\u5f53\u6211\u4eec\u8c08\u8bba\u68c0\u7d22\u589e\u5f3a\u65f6\uff0c\u6211\u4eec\u6307\u7684\u662f\u5411LLM\u63d0\u4f9b\u6709\u4ef7\u503c\u7684\u6e90\u77e5\u8bc6\u3002<\/p>\n<h2 data-id=\"heading-1\">\u83b7\u53d6\u6211\u4eec\u77e5\u8bc6\u5e93\u7684\u6570\u636e<\/h2>\n<p>\u4e3a\u4e86\u5e2e\u52a9\u6211\u4eec\u7684LLM\uff0c\u6211\u4eec\u9700\u8981\u4e3a\u5176\u63d0\u4f9b\u8bbf\u95ee\u76f8\u5173\u6e90\u77e5\u8bc6\u7684\u80fd\u529b\u3002\u4e3a\u4e86\u5b9e\u73b0\u8fd9\u4e00\u70b9\uff0c\u6211\u4eec\u9700\u8981\u521b\u5efa\u6211\u4eec\u81ea\u5df1\u7684\u77e5\u8bc6\u5e93\u3002<\/p>\n<p>\u6211\u4eec\u4ece\u4e00\u4e2a\u6570\u636e\u96c6\u5f00\u59cb\u3002\u6240\u4f7f\u7528\u7684\u6570\u636e\u96c6\u81ea\u7136\u53d6\u51b3\u4e8e\u7528\u4f8b\u3002\u5b83\u53ef\u4ee5\u662f\u7528\u4e8e\u534f\u52a9\u7f16\u5199\u4ee3\u7801\u7684\u4ee3\u7801\u6587\u6863\uff0c\u7528\u4e8e\u5185\u90e8\u804a\u5929\u673a\u5668\u4eba\u7684\u516c\u53f8\u6587\u4ef6\uff0c\u6216\u8005\u5176\u4ed6\u4efb\u4f55\u5185\u5bb9\u3002<\/p>\n<p>\u5728\u6211\u4eec\u7684\u793a\u4f8b\u4e2d\uff0c\u6211\u4eec\u5c06\u4f7f\u7528\u7ef4\u57fa\u767e\u79d1\u7684\u4e00\u4e2a\u5b50\u96c6\u3002\u4e3a\u4e86\u83b7\u53d6\u8fd9\u4e9b\u6570\u636e\uff0c\u6211\u4eec\u5c06\u4f7f\u7528Hugging Face\u6570\u636e\u96c6\uff0c\u5982\u4e0b\u6240\u793a\uff1a<\/p>\n<p>In[2]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">kotlin<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-kotlin code-block-extension-codeShowNum\" lang=\"kotlin\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from datasets <span class=\"hljs-keyword\">import<\/span> load_dataset<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-keyword\">data<\/span> = load_dataset(<span class=\"hljs-string\">\"wikipedia\"<\/span>, <span class=\"hljs-string\">\"20220301.simple\"<\/span>, split=<span class=\"hljs-string\">'train[:10000]'<\/span>)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><span class=\"hljs-keyword\">data<\/span><\/span>\n<\/code><\/pre>\n<p>Out[2]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ruby<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ruby code-block-extension-codeShowNum\" lang=\"ruby\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-title class_\">Downloading<\/span> <span class=\"hljs-symbol\">readme:<\/span>   <span class=\"hljs-number\">0<\/span><span class=\"hljs-string\">%|          |<\/span> <span class=\"hljs-number\">0.00<\/span>\/<span class=\"hljs-number\">16<\/span>.3k [<span class=\"hljs-number\">00<\/span><span class=\"hljs-symbol\">:<\/span><span class=\"hljs-number\">00<\/span>&lt;<span class=\"hljs-string\">?,<\/span> <span class=\"hljs-string\">?B<\/span>\/s]<\/span>\n<\/code><\/pre>\n<p>Out[2]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">php<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-php code-block-extension-codeShowNum\" lang=\"php\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-title function_ invoke__\">Dataset<\/span>({<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\">    <span class=\"hljs-attr\">features<\/span>: [<span class=\"hljs-string\">'id'<\/span>, <span class=\"hljs-string\">'url'<\/span>, <span class=\"hljs-string\">'title'<\/span>, <span class=\"hljs-string\">'text'<\/span>],<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\">    <span class=\"hljs-attr\">num_rows<\/span>: <span class=\"hljs-number\">10000<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\">})<\/span>\n<\/code><\/pre>\n<p>In[3]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">css<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-css code-block-extension-codeShowNum\" lang=\"css\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">data<span class=\"hljs-selector-attr\">[6]<\/span><\/span>\n<\/code><\/pre>\n<p>Out[3]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">vbnet<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-vbnet code-block-extension-codeShowNum\" lang=\"vbnet\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">{<span class=\"hljs-comment\">'id': '13',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> <span class=\"hljs-comment\">'url': 'https:\/\/simple.wikipedia.org\/wiki\/Alan%20Turing',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> <span class=\"hljs-comment\">'title': 'Alan Turing',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"> <span class=\"hljs-comment\">'text': 'Alan Mathison Turing OBE FRS (London, 23 June 1912 \u2013 Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.nnEarly life and family nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.nnEducation nTuring went to St. Michael's, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.\u201d \u2013 Alan Turing.nnThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881\u20131976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.nnEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.nnA brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.nnIn 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the \u201cTuring law\u201d grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.nnAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.nnCareer nTuring was one of the people who worked on the first computers. He created the theoretical  Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.nnTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.nnDuring World War II, Turing worked with others to break German ciphers (secret messages). He  worked for the Government Code and Cypher School (GC&amp;CS) at Bletchley Park, Britain's codebreaking centre that produced Ultra intelligence.nUsing cryptanalysis, he helped to break the codes of the Enigma machine. After that, he worked on other German codes.nnFrom 1945 to 1947, Turing worked on the design of the ACE (Automatic Computing Engine) at the National Physical Laboratory. He presented a paper on 19 February 1946. That paper was \"the first detailed design of a stored-program computer\". Although it was possible to build ACE, there were delays in starting the project. In late 1947 he returned to Cambridge for a sabbatical year. While he was at Cambridge, the Pilot ACE was built without him. It ran its first program on 10xa0May 1950.nnPrivate life nTuring was a homosexual man. In 1952, he admitted having had sex with a man in England. At that time, homosexual acts were illegal. Turing was convicted. He had to choose between going to jail and taking hormones to lower his sex drive. He decided to take the hormones. After his punishment, he became impotent. He also grew breasts.nnIn May 2012, a private member's bill was put before the House of Lords to grant Turing a statutory pardon. In July 2013, the government supported it. A royal pardon was granted on 24 December 2013.nnDeath nIn 1954, Turing died from cyanide poisoning. The cyanide came from either an apple which was poisoned with cyanide, or from water that had cyanide in it. The reason for the confusion is that the police never tested the apple for cyanide. It is also suspected that he committed suicide.nnThe treatment forced on him is now believed to be very wrong. It is against medical ethics and international laws of human rights. In August 2009, a petition asking the British Government to apologise to Turing for punishing him for being a homosexual was started. The petition received thousands of signatures. Prime Minister Gordon Brown acknowledged the petition. He called Turing's treatment \"appalling\".nnReferencesnnOther websites nJack Copeland 2012. Alan Turing: The codebreaker who saved 'millions of lives'. BBC News \/ Technology nnEnglish computer scientistsnEnglish LGBT peoplenEnglish mathematiciansnGay mennLGBT scientistsnScientists from LondonnSuicides by poisonnSuicides in the United Kingdomn1912 birthsn1954 deathsnOfficers of the Order of the British Empire'}<\/span><\/span>\n<\/code><\/pre>\n<p>\u5927\u591a\u6570\u6570\u636e\u96c6\u5c06\u5305\u542b\u5305\u542b\u5927\u91cf\u6587\u672c\u7684\u8bb0\u5f55\u3002\u56e0\u6b64\uff0c\u6211\u4eec\u901a\u5e38\u7684\u7b2c\u4e00\u4e2a\u4efb\u52a1\u662f\u6784\u5efa\u4e00\u4e2a\u9884\u5904\u7406\u7ba1\u9053\uff0c\u5c06\u90a3\u4e9b\u957f\u6587\u672c\u5206\u5272\u6210\u66f4\u7b80\u6d01\u7684\u5757\u3002<\/p>\n<h2 data-id=\"heading-2\">\u521b\u5efa\u6587\u672c\u5757<\/h2>\n<p>\u5c06\u6211\u4eec\u7684\u6587\u672c\u5206\u5272\u6210\u8f83\u5c0f\u7684\u5757\u5bf9\u4e8e\u591a\u4e2a\u539f\u56e0\u81f3\u5173\u91cd\u8981\u3002\u4e3b\u8981\u76ee\u7684\u662f\uff1a<\/p>\n<ul>\n<li>\n<p>\u63d0\u9ad8\u201c\u5d4c\u5165\u51c6\u786e\u6027\u201d &#8211; \u8fd9\u5c06\u63d0\u9ad8\u540e\u7eed\u7ed3\u679c\u7684\u76f8\u5173\u6027\u3002<\/p>\n<\/li>\n<li>\n<p>\u51cf\u5c11\u8f93\u5165\u5230\u6211\u4eecLLM\u4f5c\u4e3a\u6e90\u77e5\u8bc6\u7684\u6587\u672c\u6570\u91cf\u3002\u9650\u5236\u8f93\u5165\u53ef\u4ee5\u63d0\u9ad8LLM\u9075\u5faa\u6307\u4ee4\u7684\u80fd\u529b\uff0c\u964d\u4f4e\u751f\u6210\u6210\u672c\uff0c\u5e76\u5e2e\u52a9\u6211\u4eec\u83b7\u5f97\u66f4\u5feb\u7684\u54cd\u5e94\u3002<\/p>\n<\/li>\n<li>\n<p>\u4e3a\u7528\u6237\u63d0\u4f9b\u66f4\u7cbe\u786e\u7684\u4fe1\u606f\u6e90\uff0c\u56e0\u4e3a\u6211\u4eec\u53ef\u4ee5\u5c06\u4fe1\u606f\u6e90\u7f29\u5c0f\u5230\u66f4\u5c0f\u7684\u6587\u672c\u5757\u3002<\/p>\n<\/li>\n<li>\n<p>\u5bf9\u4e8e\u975e\u5e38\u957f\u7684\u6587\u672c\u5757\uff0c\u6211\u4eec\u5c06\u8d85\u51fa\u5d4c\u5165\u6216\u5b8c\u6210\u6a21\u578b\u7684\u6700\u5927\u4e0a\u4e0b\u6587\u7a97\u53e3\u3002\u5206\u5272\u8fd9\u4e9b\u6587\u672c\u5757\u4f7f\u5f97\u5c06\u8fd9\u4e9b\u8f83\u957f\u7684\u6587\u6863\u6dfb\u52a0\u5230\u6211\u4eec\u7684\u77e5\u8bc6\u5e93\u6210\u4e3a\u53ef\u80fd\u3002<\/p>\n<\/li>\n<\/ul>\n<p>\u8981\u521b\u5efa\u8fd9\u4e9b\u5757\uff0c\u9996\u5148\u9700\u8981\u4e00\u79cd\u6d4b\u91cf\u6587\u672c\u957f\u5ea6\u7684\u65b9\u6cd5\u3002LLMs\u4e0d\u662f\u6309\u5355\u8bcd\u6216\u5b57\u7b26\u6d4b\u91cf\u6587\u672c\u7684 &#8211; \u5b83\u4eec\u662f\u6309\u201c\u6807\u8bb0\u201d\u6765\u6d4b\u91cf\u7684\u3002<\/p>\n<p>\u6807\u8bb0\u901a\u5e38\u662f\u5355\u8bcd\u6216\u5b50\u8bcd\u7684\u5927\u5c0f\uff0c\u4e0d\u540c\u7684LLMs\u6709\u4e0d\u540c\u7684\u6807\u8bb0\u5927\u5c0f\u3002\u6807\u8bb0\u672c\u8eab\u662f\u7531\u6807\u8bb0\u5668\u6784\u5efa\u7684\u3002\u6211\u4eec\u5c06\u4f7f\u7528gpt-3.5-turbo\u4f5c\u4e3a\u6211\u4eec\u7684\u5b8c\u6210\u6a21\u578b\uff0c\u5e76\u4e14\u6211\u4eec\u53ef\u4ee5\u50cf\u8fd9\u6837\u521d\u59cb\u5316\u8be5\u6a21\u578b\u7684\u6807\u8bb0\u5668\uff1a<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">import tiktoken  <span class=\"hljs-comment\"># !pip install tiktoken<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">tokenizer<\/span> = tiktoken.get_encoding(<span class=\"hljs-string\">'p50k_base'<\/span>)<\/span>\n<\/code><\/pre>\n<p>\u4f7f\u7528\u6807\u8bb0\u5668\uff0c\u6211\u4eec\u53ef\u4ee5\u4ece\u7eaf\u6587\u672c\u521b\u5efa\u6807\u8bb0\u5e76\u8ba1\u7b97\u6807\u8bb0\u7684\u6570\u91cf\u3002\u6211\u4eec\u5c06\u628a\u8fd9\u4e2a\u8fc7\u7a0b\u5c01\u88c5\u5230\u4e00\u4e2a\u540d\u4e3atiktoken_len\u7684\u51fd\u6570\u4e2d\uff1a<\/p>\n<p>In[28]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">arduino<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-arduino code-block-extension-codeShowNum\" lang=\"arduino\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-meta\"># create the length function<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><span class=\"hljs-function\">def <span class=\"hljs-title\">tiktoken_len<\/span><span class=\"hljs-params\">(text)<\/span>:<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\">    tokens = tokenizer.<span class=\"hljs-built_in\">encode<\/span>(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\">        text,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\">        disallowed_special=()<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">    )<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">    <span class=\"hljs-keyword\">return<\/span> <span class=\"hljs-built_in\">len<\/span>(tokens)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\"><span class=\"hljs-built_in\">tiktoken_len<\/span>(<span class=\"hljs-string\">\"hello I am a chunk of text and using the tiktoken_len function \"<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"10\">             <span class=\"hljs-string\">\"we can find the length of this chunk of text in tokens\"<\/span>)<\/span>\n<\/code><\/pre>\n<p>Out[28]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\"><\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs code-block-extension-codeShowNum\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">28<\/span>\n<\/code><\/pre>\n<p>\u6709\u4e86\u6211\u4eec\u7684\u6807\u8bb0\u8ba1\u6570\u51fd\u6570\u51c6\u5907\u597d\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u521d\u59cb\u5316\u4e00\u4e2aLangChain RecursiveCharacterTextSplitter\u5bf9\u8c61\u3002\u8fd9\u4e2a\u5bf9\u8c61\u5c06\u5141\u8bb8\u6211\u4eec\u5c06\u6587\u672c\u5206\u5272\u6210\u4e0d\u8d85\u8fc7\u6211\u4eec\u901a\u8fc7chunk_size\u53c2\u6570\u6307\u5b9a\u7684\u957f\u5ea6\u7684\u5757\u3002<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from langchain.text_splitter import RecursiveCharacterTextSplitter<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">text_splitter<\/span> = RecursiveCharacterTextSplitter(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\">    <span class=\"hljs-attr\">chunk_size<\/span>=<span class=\"hljs-number\">400<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\">    <span class=\"hljs-attr\">chunk_overlap<\/span>=<span class=\"hljs-number\">20<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">    <span class=\"hljs-attr\">length_function<\/span>=tiktoken_len,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">    <span class=\"hljs-attr\">separators<\/span>=[<span class=\"hljs-string\">\"nn\"<\/span>, <span class=\"hljs-string\">\"n\"<\/span>, <span class=\"hljs-string\">\" \"<\/span>, <span class=\"hljs-string\">\"\"<\/span>]<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\">)<\/span>\n<\/code><\/pre>\n<p>\u73b0\u5728\u6211\u4eec\u53ef\u4ee5\u8fd9\u6837\u5206\u5272\u6587\u672c\uff1a<\/p>\n<p>In[6]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">css<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-css code-block-extension-codeShowNum\" lang=\"css\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">chunks = text_splitter<span class=\"hljs-selector-class\">.split_text<\/span>(data<span class=\"hljs-selector-attr\">[6]<\/span><span class=\"hljs-selector-attr\">[<span class=\"hljs-string\">'text'<\/span>]<\/span>)<span class=\"hljs-selector-attr\">[:3]<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\">chunks<\/span>\n<\/code><\/pre>\n<p>Out[6]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">vbnet<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-vbnet code-block-extension-codeShowNum\" lang=\"vbnet\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">[<span class=\"hljs-comment\">'Alan Mathison Turing OBE FRS (London, 23 June 1912 \u2013 Wilmslow, Cheshire, 7 June 1954) was an English mathematician and computer scientist. He was born in Maida Vale, London.nnEarly life and family nAlan Turing was born in Maida Vale, London on 23 June 1912. His father was part of a family of merchants from Scotland. His mother, Ethel Sara, was the daughter of an engineer.nnEducation nTuring went to St. Michael's, a school at 20 Charles Road, St Leonards-on-sea, when he was five years old.n\"This is only a foretaste of what is to come, and only the shadow of what is going to be.\u201d \u2013 Alan Turing.nnThe Stoney family were once prominent landlords, here in North Tipperary. His mother Ethel Sara Stoney (1881\u20131976) was daughter of Edward Waller Stoney (Borrisokane, North Tipperary) and Sarah Crawford (Cartron Abbey, Co. Longford); Protestant Anglo-Irish gentry.nnEducated in Dublin at Alexandra School and College; on October 1st 1907 she married Julius Mathison Turing, latter son of Reverend John Robert Turing and Fanny Boyd, in Dublin. Born on June 23rd 1912, Alan Turing would go on to be regarded as one of the greatest figures of the twentieth century.nnA brilliant mathematician and cryptographer Alan was to become the founder of modern-day computer science and artificial intelligence; designing a machine at Bletchley Park to break secret Enigma encrypted messages used by the Nazi German war machine to protect sensitive commercial, diplomatic and military communications during World War 2. Thus, Turing made the single biggest contribution to the Allied victory in the war against Nazi Germany, possibly saving the lives of an estimated 2 million people, through his effort in shortening World War II.',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> <span class=\"hljs-comment\">'In 2013, almost 60 years later, Turing received a posthumous Royal Pardon from Queen Elizabeth II. Today, the \u201cTuring law\u201d grants an automatic pardon to men who died before the law came into force, making it possible for living convicted gay men to seek pardons for offences now no longer on the statute book.nnAlas, Turing accidentally or otherwise lost his life in 1954, having been subjected by a British court to chemical castration, thus avoiding a custodial sentence. He is known to have ended his life at the age of 41 years, by eating an apple laced with cyanide.nnCareer nTuring was one of the people who worked on the first computers. He created the theoretical  Turing machine in 1936. The machine was imaginary, but it included the idea of a computer program.nnTuring was interested in artificial intelligence. He proposed the Turing test, to say when a machine could be called \"intelligent\". A computer could be said to \"think\" if a human talking with it could not tell it was a machine.nnDuring World War II, Turing worked with others to break German ciphers (secret messages). He  worked for the Government Code and Cypher School (GC&amp;CS) at Bletchley Park, Britain's codebreaking centre that produced Ultra intelligence.nUsing cryptanalysis, he helped to break the codes of the Enigma machine. After that, he worked on other German codes.',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> <span class=\"hljs-comment\">'From 1945 to 1947, Turing worked on the design of the ACE (Automatic Computing Engine) at the National Physical Laboratory. He presented a paper on 19 February 1946. That paper was \"the first detailed design of a stored-program computer\". Although it was possible to build ACE, there were delays in starting the project. In late 1947 he returned to Cambridge for a sabbatical year. While he was at Cambridge, the Pilot ACE was built without him. It ran its first program on 10xa0May 1950.nnPrivate life nTuring was a homosexual man. In 1952, he admitted having had sex with a man in England. At that time, homosexual acts were illegal. Turing was convicted. He had to choose between going to jail and taking hormones to lower his sex drive. He decided to take the hormones. After his punishment, he became impotent. He also grew breasts.nnIn May 2012, a private member's bill was put before the House of Lords to grant Turing a statutory pardon. In July 2013, the government supported it. A royal pardon was granted on 24 December 2013.nnDeath nIn 1954, Turing died from cyanide poisoning. The cyanide came from either an apple which was poisoned with cyanide, or from water that had cyanide in it. The reason for the confusion is that the police never tested the apple for cyanide. It is also suspected that he committed suicide.nnThe treatment forced on him is now believed to be very wrong. It is against medical ethics and international laws of human rights. In August 2009, a petition asking the British Government to apologise to Turing for punishing him for being a homosexual was started. The petition received thousands of signatures. Prime Minister Gordon Brown acknowledged the petition. He called Turing's treatment \"appalling\".nnReferencesnnOther websites nJack Copeland 2012. Alan Turing: The codebreaker who saved 'millions of lives'. BBC News \/ Technology']<\/span><\/span>\n<\/code><\/pre>\n<p>\u8fd9\u4e9b\u5757\u90fd\u6ca1\u6709\u8d85\u8fc7\u6211\u4eec\u4e4b\u524d\u8bbe\u7f6e\u7684400\u5757\u5927\u5c0f\u9650\u5236\uff1a<\/p>\n<p>In[7]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-built_in\">tiktoken_len<\/span>(chunks[<span class=\"hljs-number\">0<\/span>]), <span class=\"hljs-built_in\">tiktoken_len<\/span>(chunks[<span class=\"hljs-number\">1<\/span>]), <span class=\"hljs-built_in\">tiktoken_len<\/span>(chunks[<span class=\"hljs-number\">2<\/span>])<\/span>\n<\/code><\/pre>\n<p>Out[7]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">(<span class=\"hljs-number\">397<\/span>, <span class=\"hljs-number\">304<\/span>, <span class=\"hljs-number\">399<\/span>)<\/span>\n<\/code><\/pre>\n<p>\u4f7f\u7528\u6587\u672c\u5206\u5272\u5668\uff0c\u6211\u4eec\u83b7\u5f97\u4e86\u9002\u5f53\u5927\u5c0f\u7684\u6587\u672c\u5757\u3002\u7a0d\u540e\u5728\u7d22\u5f15\u8fc7\u7a0b\u4e2d\uff0c\u6211\u4eec\u5c06\u4f7f\u7528\u8fd9\u4e2a\u529f\u80fd\u3002\u73b0\u5728\uff0c\u8ba9\u6211\u4eec\u6765\u770b\u770b\u5d4c\u5165\u3002<\/p>\n<h2 data-id=\"heading-3\">\u521b\u5efa\u5d4c\u5165<\/h2>\n<p>\u77e2\u91cf\u5d4c\u5165\u5bf9\u4e8e\u68c0\u7d22LLM\u7684\u76f8\u5173\u4e0a\u4e0b\u6587\u81f3\u5173\u91cd\u8981\u3002\u6211\u4eec\u5c06\u5e0c\u671b\u5b58\u50a8\u5728\u6211\u4eec\u7684\u77e5\u8bc6\u5e93\u4e2d\u7684\u6587\u672c\u5757\u8fdb\u884c\u7f16\u7801\uff0c\u5c06\u6bcf\u4e2a\u5757\u7f16\u7801\u6210\u4e00\u4e2a\u77e2\u91cf\u5d4c\u5165\u3002<\/p>\n<p>\u8fd9\u4e9b\u5d4c\u5165\u53ef\u4ee5\u5145\u5f53\u6bcf\u4e2a\u6587\u672c\u5757\u542b\u4e49\u7684\u201c\u6570\u503c\u8868\u793a\u201d\u3002\u8fd9\u662f\u53ef\u80fd\u7684\uff0c\u56e0\u4e3a\u6211\u4eec\u4f7f\u7528\u53e6\u4e00\u4e2a\u5df2\u7ecf\u5b66\u4f1a\u5c06\u4eba\u7c7b\u53ef\u8bfb\u6587\u672c\u8f6c\u5316\u4e3aAI\u53ef\u8bfb\u5d4c\u5165\u7684AI\u8bed\u8a00\u6a21\u578b\u6765\u521b\u5efa\u8fd9\u4e9b\u5d4c\u5165\u3002<\/p>\n<p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/www.nicekj.com\/wp-content\/uploads\/replace\/0f1c39e0b9ad85cf7ef4a878f2eb245b.png\" alt=\"image.png\" \/><\/figure>\n<\/p>\n<p>\u7136\u540e\uff0c\u6211\u4eec\u5c06\u8fd9\u4e9b\u5d4c\u5165\u5b58\u50a8\u5728\u6211\u4eec\u7684\u77e2\u91cf\u6570\u636e\u5e93\u4e2d\uff08\u7a0d\u540e\u4f1a\u8be6\u7ec6\u4ecb\u7ecd\uff09\uff0c\u5e76\u901a\u8fc7\u5728\u77e2\u91cf\u7a7a\u95f4\u4e2d\u8ba1\u7b97\u5d4c\u5165\u4e4b\u95f4\u7684\u8ddd\u79bb\u6765\u627e\u5230\u5177\u6709\u76f8\u4f3c\u542b\u4e49\u7684\u6587\u672c\u5757\u3002<\/p>\n<p>\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/www.nicekj.com\/wp-content\/uploads\/replace\/cf9c7a5c2f3fa28babcea80197228d43.png\" alt=\"image.png\" \/><\/figure>\n<\/p>\n<p>\u6211\u4eec\u5c06\u4f7f\u7528\u7684\u5d4c\u5165\u6a21\u578b\u662f\u53e6\u4e00\u4e2aOpenAI\u6a21\u578b\uff0c\u79f0\u4e3atext-embedding-ada-002\u3002\u6211\u4eec\u53ef\u4ee5\u50cf\u8fd9\u6837\u901a\u8fc7LangChain\u521d\u59cb\u5316\u5b83\uff1a<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from langchain.embeddings.openai import OpenAIEmbeddings<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">model_name<\/span> = <span class=\"hljs-string\">'text-embedding-ada-002'<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\"><span class=\"hljs-attr\">embed<\/span> = OpenAIEmbeddings(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">    <span class=\"hljs-attr\">document_model_name<\/span>=model_name,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">    <span class=\"hljs-attr\">query_model_name<\/span>=model_name,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\">    <span class=\"hljs-attr\">openai_api_key<\/span>=OPENAI_API_KEY<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\">)<\/span>\n<\/code><\/pre>\n<p>\u73b0\u5728\u6211\u4eec\u53ef\u4ee5\u5d4c\u5165\u6211\u4eec\u7684\u6587\u672c\uff1a<\/p>\n<p>In[10]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">texts = <span class=\"hljs-selector-attr\">[    <span class=\"hljs-string\">'this is the first chunk of text'<\/span>,    <span class=\"hljs-string\">'then another second chunk of text is here'<\/span>]<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\">res = embed<span class=\"hljs-selector-class\">.embed_documents<\/span>(texts)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><span class=\"hljs-built_in\">len<\/span>(res), <span class=\"hljs-built_in\">len<\/span>(res[<span class=\"hljs-number\">0<\/span>])<\/span>\n<\/code><\/pre>\n<p>Out[10]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">(<span class=\"hljs-number\">2<\/span>, <span class=\"hljs-number\">1536<\/span>)<\/span>\n<\/code><\/pre>\n<p>\u4ece\u8fd9\u91cc\uff0c\u6211\u4eec\u5f97\u5230\u4e86\u4e24\u4e2a\u5d4c\u5165\uff0c\u56e0\u4e3a\u6211\u4eec\u4f20\u5165\u4e86\u4e24\u4e2a\u6587\u672c\u5757\u3002\u6bcf\u4e2a\u5d4c\u5165\u90fd\u662f\u4e00\u4e2a1536\u7ef4\u7684\u5411\u91cf\u3002\u8fd9\u4e2a\u7ef4\u5ea6\u53ea\u662ftext-embedding-ada-002\u7684\u8f93\u51fa\u7ef4\u5ea6\u3002<\/p>\n<p>\u6709\u4e86\u8fd9\u4e9b\uff0c\u6211\u4eec\u62e5\u6709\u4e86\u6211\u4eec\u7684\u6570\u636e\u96c6\u3001\u6587\u672c\u5206\u5272\u5668\u548c\u5d4c\u5165\u6a21\u578b\u3002\u6211\u4eec\u5177\u5907\u4e86\u5f00\u59cb\u6784\u5efa\u77e5\u8bc6\u5e93\u6240\u9700\u7684\u4e00\u5207\u3002<\/p>\n<h2 data-id=\"heading-4\">\u5411\u91cf\u6570\u636e\u5e93<\/h2>\n<p>\u77e2\u91cf\u6570\u636e\u5e93\u662f\u4e00\u79cd\u77e5\u8bc6\u5e93\u7c7b\u578b\uff0c\u5b83\u5141\u8bb8\u6211\u4eec\u6269\u5c55\u5bf9\u6570\u5341\u4ebf\u6761\u8bb0\u5f55\u7684\u76f8\u4f3c\u5d4c\u5165\u7684\u641c\u7d22\uff0c\u901a\u8fc7\u6dfb\u52a0\u3001\u66f4\u65b0\u6216\u5220\u9664\u8bb0\u5f55\u6765\u7ba1\u7406\u6211\u4eec\u7684\u77e5\u8bc6\u5e93\uff0c\u751a\u81f3\u53ef\u4ee5\u6267\u884c\u8bf8\u5982\u8fc7\u6ee4\u7b49\u64cd\u4f5c\u3002<\/p>\n<p>\u6211\u4eec\u5c06\u4f7f\u7528Pinecone\u77e2\u91cf\u6570\u636e\u5e93\u3002\u8981\u4f7f\u7528\u5b83\uff0c\u6211\u4eec\u9700\u8981\u4e00\u4e2a\u514d\u8d39\u7684API\u5bc6\u94a5\u3002\u7136\u540e\uff0c\u6211\u4eec\u53ef\u4ee5\u50cf\u8fd9\u6837\u521d\u59cb\u5316\u6211\u4eec\u7684\u6570\u636e\u5e93\u7d22\u5f15\uff1a<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">import pinecone<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">index_name<\/span> = <span class=\"hljs-string\">'langchain-retrieval-augmentation'<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\">pinecone.init(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">        <span class=\"hljs-attr\">api_key<\/span>=<span class=\"hljs-string\">\"YOUR_API_KEY\"<\/span>,  <span class=\"hljs-comment\"># find api key in console at app.pinecone.io<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">        <span class=\"hljs-attr\">environment<\/span>=<span class=\"hljs-string\">\"YOUR_ENV\"<\/span>  <span class=\"hljs-comment\"># find next to api key in console<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\">)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"10\"><span class=\"hljs-comment\"># we create a new index<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"11\">pinecone.create_index(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"12\">        <span class=\"hljs-attr\">name<\/span>=index_name,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"13\">        <span class=\"hljs-attr\">metric<\/span>=<span class=\"hljs-string\">'dotproduct'<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"14\">        <span class=\"hljs-attr\">dimension<\/span>=len(res[<span class=\"hljs-number\">0<\/span>]) <span class=\"hljs-comment\"># 1536 dim of text-embedding-ada-002<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"15\">)<\/span>\n<\/code><\/pre>\n<p>\u7136\u540e\uff0c\u6211\u4eec\u8fde\u63a5\u5230\u65b0\u7684\u7d22\u5f15\uff1a<\/p>\n<p>In[12]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">diff<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-diff code-block-extension-codeShowNum\" lang=\"diff\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-comment\">index = pinecone.GRPCIndex(index_name)<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-comment\">index.describe_index_stats()<\/span><\/span>\n<\/code><\/pre>\n<p>Out[12]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">yaml<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-yaml code-block-extension-codeShowNum\" lang=\"yaml\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">{<span class=\"hljs-attr\">'dimension':<\/span> <span class=\"hljs-number\">1536<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> <span class=\"hljs-attr\">'index_fullness':<\/span> <span class=\"hljs-number\">0.0<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> <span class=\"hljs-attr\">'namespaces':<\/span> {},<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"> <span class=\"hljs-attr\">'total_vector_count':<\/span> <span class=\"hljs-number\">0<\/span>}<\/span>\n<\/code><\/pre>\n<p>\u6211\u4eec\u4f1a\u770b\u5230\u65b0\u7684Pinecone\u7d22\u5f15\u7684total_vector_count\u4e3a0\uff0c\u56e0\u4e3a\u6211\u4eec\u8fd8\u6ca1\u6709\u6dfb\u52a0\u4efb\u4f55\u5411\u91cf\u3002\u6211\u4eec\u63a5\u4e0b\u6765\u7684\u4efb\u52a1\u662f\u5b8c\u6210\u8fd9\u4e00\u6b65\u3002<\/p>\n<p>\u7d22\u5f15\u8fc7\u7a0b\u5305\u62ec\u904d\u5386\u6211\u4eec\u5e0c\u671b\u6dfb\u52a0\u5230\u77e5\u8bc6\u5e93\u4e2d\u7684\u6570\u636e\uff0c\u521b\u5efaID\u3001\u5d4c\u5165\u548c\u5143\u6570\u636e\uff0c\u7136\u540e\u5c06\u5b83\u4eec\u6dfb\u52a0\u5230\u7d22\u5f15\u4e2d\u3002<\/p>\n<p>\u6211\u4eec\u53ef\u4ee5\u5206\u6279\u8fdb\u884c\u6b64\u64cd\u4f5c\u4ee5\u52a0\u901f\u8fc7\u7a0b\u3002<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from tqdm.auto import tqdm<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\">from uuid import uuid4<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><span class=\"hljs-attr\">batch_limit<\/span> = <span class=\"hljs-number\">100<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\"><span class=\"hljs-attr\">texts<\/span> = []<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\"><span class=\"hljs-attr\">metadatas<\/span> = []<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\">for i, record in enumerate(tqdm(data)):<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"10\">    <span class=\"hljs-comment\"># first get metadata fields for this record<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"11\">    <span class=\"hljs-attr\">metadata<\/span> = {<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"12\">        'wiki-id': str(record<span class=\"hljs-section\">['id']<\/span>),<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"13\">        'source': record<span class=\"hljs-section\">['url']<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"14\">        'title': record<span class=\"hljs-section\">['title']<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"15\">    }<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"16\">    <span class=\"hljs-comment\"># now we create chunks from the record text<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"17\">    <span class=\"hljs-attr\">record_texts<\/span> = text_splitter.split_text(record[<span class=\"hljs-string\">'text'<\/span>])<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"18\">    <span class=\"hljs-comment\"># create individual metadata dicts for each chunk<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"19\">    <span class=\"hljs-attr\">record_metadatas<\/span> = [{<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"20\">        <span class=\"hljs-string\">\"chunk\"<\/span>: j, <span class=\"hljs-string\">\"text\"<\/span>: text, **metadata<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"21\">    } for j, text in enumerate(record_texts)]<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"22\">    <span class=\"hljs-comment\"># append these to current batches<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"23\">    texts.extend(record_texts)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"24\">    metadatas.extend(record_metadatas)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"25\">    <span class=\"hljs-comment\"># if we have reached the batch_limit we can add texts<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"26\">    if len(texts) &gt;= batch_limit:<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"27\">        <span class=\"hljs-attr\">ids<\/span> = [str(uuid4()) for _ in range(len(texts))]<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"28\">        <span class=\"hljs-attr\">embeds<\/span> = embed.embed_documents(texts)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"29\">        index.upsert(<span class=\"hljs-attr\">vectors<\/span>=zip(ids, embeds, metadatas))<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"30\">        <span class=\"hljs-attr\">texts<\/span> = []<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"31\">        <span class=\"hljs-attr\">metadatas<\/span> = []<\/span>\n<\/code><\/pre>\n<p>\u73b0\u5728\u6211\u4eec\u5df2\u7ecf\u5b8c\u6210\u4e86\u7d22\u5f15\u3002\u8981\u68c0\u67e5\u7d22\u5f15\u4e2d\u8bb0\u5f55\u7684\u6570\u91cf\uff0c\u6211\u4eec\u518d\u6b21\u8c03\u7528describe_index_stats\uff1a<\/p>\n<p>In[14]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">diff<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-diff code-block-extension-codeShowNum\" lang=\"diff\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-comment\">index.describe_index_stats()<\/span><\/span>\n<\/code><\/pre>\n<p>Out[14]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">bash<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-bash code-block-extension-codeShowNum\" lang=\"bash\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">{<span class=\"hljs-string\">'dimension'<\/span>: 1536,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> <span class=\"hljs-string\">'index_fullness'<\/span>: 0.1,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> <span class=\"hljs-string\">'namespaces'<\/span>: {<span class=\"hljs-string\">''<\/span>: {<span class=\"hljs-string\">'vector_count'<\/span>: 27437}},<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"> <span class=\"hljs-string\">'total_vector_count'<\/span>: 27437}<\/span>\n<\/code><\/pre>\n<p>\u6211\u4eec\u7684\u7d22\u5f15\u5305\u542b\u5927\u7ea627,000\u6761\u8bb0\u5f55\u3002\u6b63\u5982\u4e4b\u524d\u63d0\u5230\u7684\uff0c\u6211\u4eec\u53ef\u4ee5\u5c06\u8fd9\u4e2a\u6570\u91cf\u6269\u5c55\u5230\u6570\u5341\u4ebf\uff0c\u4f46\u5bf9\u4e8e\u6211\u4eec\u7684\u793a\u4f8b\u6765\u8bf4\uff0c27,000\u8db3\u591f\u4e86\u3002<\/p>\n<h1 data-id=\"heading-5\">LangChain\u77e2\u91cf\u5b58\u50a8\u548c\u67e5\u8be2<\/h1>\n<p>\u6211\u4eec\u72ec\u7acb\u4e8eLangChain\u6784\u5efa\u6211\u4eec\u7684\u7d22\u5f15\u3002\u8fd9\u662f\u56e0\u4e3a\u8fd9\u662f\u4e00\u4e2a\u7b80\u5355\u7684\u8fc7\u7a0b\uff0c\u800c\u4e14\u76f4\u63a5\u4f7f\u7528Pinecone\u5ba2\u6237\u7aef\u66f4\u5feb\u3002\u7136\u800c\uff0c\u6211\u4eec\u5373\u5c06\u56de\u5230LangChain\uff0c\u6240\u4ee5\u6211\u4eec\u5e94\u8be5\u901a\u8fc7LangChain\u5e93\u91cd\u65b0\u8fde\u63a5\u5230\u6211\u4eec\u7684\u7d22\u5f15\u3002<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from langchain.vectorstores import Pinecone<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">text_field<\/span> = <span class=\"hljs-string\">\"text\"<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\"><span class=\"hljs-comment\"># switch back to normal index for langchain<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\"><span class=\"hljs-attr\">index<\/span> = pinecone.Index(index_name)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\"><span class=\"hljs-attr\">vectorstore<\/span> = Pinecone(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\">    index, embed.embed_query, text_field<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"10\">)<\/span>\n<\/code><\/pre>\n<p>\u6211\u4eec\u53ef\u4ee5\u4f7f\u7528\u76f8\u4f3c\u6027\u641c\u7d22\u65b9\u6cd5\u76f4\u63a5\u8fdb\u884c\u67e5\u8be2\uff0c\u8fd4\u56de\u6587\u672c\u5757\uff0c\u800c\u65e0\u9700LLM\u751f\u6210\u54cd\u5e94\u3002<\/p>\n<p>In[16]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-attr\">query<\/span> = <span class=\"hljs-string\">\"who was Benito Mussolini?\"<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\">vectorstore.similarity_search(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\">    query,  <span class=\"hljs-comment\"># our search query<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\">    <span class=\"hljs-attr\">k<\/span>=<span class=\"hljs-number\">3<\/span>  <span class=\"hljs-comment\"># return 3 most relevant docs<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">)<\/span>\n<\/code><\/pre>\n<p>Out[16]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">vbnet<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-vbnet code-block-extension-codeShowNum\" lang=\"vbnet\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">[Document(page_content=<span class=\"hljs-comment\">'Benito Amilcare Andrea Mussolini KSMOM GCTE (29 July 1883 \u2013 28 April 1945) was an Italian politician and journalist. He was also the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party.nnBiographynnEarly lifenBenito Mussolini was named after Benito Juarez, a Mexican opponent of the political power of the Roman Catholic Church, by his anticlerical (a person who opposes the political interference of the Roman Catholic Church in secular affairs) father. Mussolini's father was a blacksmith. Before being involved in politics, Mussolini was a newspaper editor (where he learned all his propaganda skills) and elementary school teacher.nnAt first, Mussolini was a socialist, but when he wanted Italy to join the First World War, he was thrown out of the socialist party. He 'invented' a new ideology, Fascism, much out of Nationalistxa0and Conservative views.nnRise to power and becoming dictatornIn 1922, he took power by having a large group of men, \"Black Shirts,\" march on Rome and threaten to take over the government. King Vittorio Emanuele III gave in, allowed him to form a government, and made him prime minister. In the following five years, he gained power, and in 1927 created the OVRA, his personal secret police force. Using the agency to arrest, scare, or murder people against his regime, Mussolini was dictatorxa0of Italy by the end of 1927. Only the King and his own Fascist party could challenge his power.', lookup_str='', metadata={'chunk': 0.0, 'source': 'https:\/\/simple.wikipedia.org\/wiki\/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, lookup_index=0),<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> Document(page_content=<span class=\"hljs-comment\">'Fascism as practiced by MussolininMussolini's form of Fascism, \"Italian Fascism\"- unlike Nazism, the racist ideology that Adolf Hitler followed- was different and less destructive than Hitler's. Although a believer in the superiority of the Italian nation and national unity, Mussolini, unlike Hitler, is quoted \"Race? It is a feeling, not a reality. Nothing will ever make me believe that biologically pure races can be shown to exist today\".nnMussolini wanted Italy to become a new Roman Empire. In 1923, he attacked the island of Corfu, and in 1924, he occupied the city state of Fiume. In 1935, he attacked the African country Abyssinia (now called Ethiopia). His forces occupied it in 1936. Italy was thrown out of the League of Nations because of this aggression. In 1939, he occupied the country Albania. In 1936, Mussolini signed an alliance with Adolf Hitler, the dictator of Germany.nnFall from power and deathnIn 1940, he sent Italy into the Second World War on the side of the Axis countries. Mussolini attacked Greece, but he failed to conquer it. In 1943, the Allies landed in Southern Italy. The Fascist party and King Vittorio Emanuel III deposed Mussolini and put him in jail, but he was set free by the Germans, who made him ruler of the Italian Social Republic puppet state which was in a small part of Central Italy. When the war was almost over, Mussolini tried to escape to Switzerland with his mistress, Clara Petacci, but they were both captured and shot by partisans. Mussolini's dead body was hanged upside-down, together with his mistress and some of Mussolini's helpers, on a pole at a gas station in the village of Millan, which is near the border  between Italy and Switzerland.', lookup_str='', metadata={'chunk': 1.0, 'source': 'https:\/\/simple.wikipedia.org\/wiki\/Benito%20Mussolini', 'title': 'Benito Mussolini', 'wiki-id': '6754'}, lookup_index=0),<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> Document(page_content=<span class=\"hljs-comment\">'Fascist Italy nIn 1922, a new Italian government started. It was ruled by Benito Mussolini, the leader of Fascism in Italy. He became head of government and dictator, calling himself \"Il Duce\" (which means \"leader\" in Italian). He became friends with German dictator Adolf Hitler. Germany, Japan, and Italy became the Axis Powers. In 1940, they entered World War II together against France, Great Britain, and later the Soviet Union. During the war, Italy controlled most of the Mediterranean Sea.nnOn July 25, 1943, Mussolini was removed by the Great Council of Fascism. On September 8, 1943, Badoglio said that the war as an ally of Germany was ended. Italy started fighting as an ally of France and the UK, but Italian soldiers did not know whom to shoot. In Northern Italy, a movement called Resistenza started to fight against the German invaders. On April 25, 1945, much of Italy became free, while Mussolini tried to make a small Northern Italian fascist state called the Republic of Sal\u00f2. The fascist state failed and Mussolini tried to flee to Switzerland and escape to Francoist Spain, but he was captured by Italian partisans. On 28 April 1945 Mussolini was executed by a partisan.nnAfter World War Two nnThe state became a republic on June 2, 1946. For the first time, women were able to vote. Italian people ended the Savoia dynasty and adopted a republic government.nnIn February 1947, Italy signed a peace treaty with the Allies. They lost all the colonies and some territorial areas (Istria and parts of Dalmatia).nnSince then Italy has joined NATO and the European Community (as a founding member). It is one of the seven biggest industrial economies in the world.nnTransportation nnThe railway network in Italy totals . It is the 17th longest in the world. High speed trains include ETR-class trains which travel at .', lookup_str='', metadata={'chunk': 5.0, 'source': 'https:\/\/simple.wikipedia.org\/wiki\/Italy', 'title': 'Italy', 'wiki-id': '363'}, lookup_index=0)]<\/span><\/span>\n<\/code><\/pre>\n<p>\u6240\u6709\u8fd9\u4e9b\u90fd\u662f\u76f8\u5173\u7684\u7ed3\u679c\uff0c\u544a\u8bc9\u6211\u4eec\u6211\u4eec\u7684\u7cfb\u7edf\u7684\u68c0\u7d22\u7ec4\u4ef6\u6b63\u5728\u8fd0\u884c\u3002\u4e0b\u4e00\u6b65\u662f\u5c06\u6211\u4eec\u7684LLM\u6dfb\u52a0\u8fdb\u6765\uff0c\u4ee5\u751f\u6210\u6027\u5730\u56de\u7b54\u6211\u4eec\u7684\u95ee\u9898\uff0c\u4f7f\u7528\u5728\u8fd9\u4e9b\u68c0\u7d22\u5230\u7684\u4e0a\u4e0b\u6587\u4e2d\u63d0\u4f9b\u7684\u4fe1\u606f\u3002<\/p>\n<h2 data-id=\"heading-6\">\u751f\u6210\u5f0f\u95ee\u7b54<\/h2>\n<p>\u5728\u751f\u6210\u5f0f\u95ee\u7b54\uff08GQA\uff09\u4e2d\uff0c\u6211\u4eec\u5c06\u95ee\u9898\u4f20\u9012\u7ed9LLM\uff0c\u4f46\u6307\u793a\u5b83\u57fa\u4e8e\u4ece\u6211\u4eec\u7684\u77e5\u8bc6\u5e93\u8fd4\u56de\u7684\u4fe1\u606f\u6765\u751f\u6210\u7b54\u6848\u3002\u6211\u4eec\u53ef\u4ee5\u5728LangChain\u4e2d\u8f7b\u677e\u5730\u4f7f\u7528RetrievalQA\u94fe\u6765\u5b9e\u73b0\u8fd9\u4e00\u70b9\u3002<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from langchain.chat_models import ChatOpenAI<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\">from langchain.chains import RetrievalQA<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\"><span class=\"hljs-comment\"># completion llm<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\"><span class=\"hljs-attr\">llm<\/span> = ChatOpenAI(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">    <span class=\"hljs-attr\">openai_api_key<\/span>=OPENAI_API_KEY,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">    <span class=\"hljs-attr\">model_name<\/span>=<span class=\"hljs-string\">'gpt-3.5-turbo'<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"8\">    <span class=\"hljs-attr\">temperature<\/span>=<span class=\"hljs-number\">0.0<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"9\">)<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"10\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"11\"><span class=\"hljs-attr\">qa<\/span> = RetrievalQA.from_chain_type(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"12\">    <span class=\"hljs-attr\">llm<\/span>=llm,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"13\">    <span class=\"hljs-attr\">chain_type<\/span>=<span class=\"hljs-string\">\"stuff\"<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"14\">    <span class=\"hljs-attr\">retriever<\/span>=vectorstore.as_retriever()<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"15\">)<\/span>\n<\/code><\/pre>\n<p>\u8ba9\u6211\u4eec\u5c1d\u8bd5\u4e00\u4e0b\u4e4b\u524d\u7684\u67e5\u8be2\uff1a<\/p>\n<p>In[22]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">qa<span class=\"hljs-selector-class\">.run<\/span>(query)<\/span>\n<\/code><\/pre>\n<p>Out[22]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">vbnet<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-vbnet code-block-extension-codeShowNum\" lang=\"vbnet\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-comment\">'Benito Mussolini was an Italian politician and journalist who served as the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and invented the ideology of Fascism. Mussolini was a dictator of Italy by the end of 1927, and his form of Fascism, \"Italian Fascism,\" was different and less destructive than Hitler's Nazism. Mussolini wanted Italy to become a new Roman Empire and attacked several countries, including Abyssinia (now called Ethiopia) and Greece. He was removed from power in 1943 and was executed by Italian partisans in 1945.'<\/span><\/span>\n<\/code><\/pre>\n<p>\u8fd9\u6b21\u6211\u4eec\u5f97\u5230\u7684\u54cd\u5e94\u662f\u7531\u6211\u4eec\u7684gpt-3.5-turbo LLM\u6839\u636e\u4ece\u6211\u4eec\u7684\u77e2\u91cf\u6570\u636e\u5e93\u4e2d\u68c0\u7d22\u5230\u7684\u4fe1\u606f\u751f\u6210\u7684\u3002<\/p>\n<p>\u5c3d\u7ba1\u5982\u6b64\uff0c\u6211\u4eec\u4ecd\u7136\u65e0\u6cd5\u5b8c\u5168\u9632\u6b62\u6a21\u578b\u751f\u6210\u4ee4\u4eba\u4fe1\u670d\u4f46\u865a\u5047\u7684\u5e7b\u89c9\uff0c\u8fd9\u79cd\u60c5\u51b5\u53ef\u80fd\u4f1a\u53d1\u751f\uff0c\u800c\u4e14\u4e0d\u592a\u53ef\u80fd\u5b8c\u5168\u6d88\u9664\u8fd9\u4e2a\u95ee\u9898\u3002\u7136\u800c\uff0c\u6211\u4eec\u53ef\u4ee5\u91c7\u53d6\u66f4\u591a\u63aa\u65bd\u6765\u63d0\u9ad8\u6211\u4eec\u5bf9\u63d0\u4f9b\u7684\u7b54\u6848\u7684\u4fe1\u4efb\u5ea6\u3002<\/p>\n<p>\u4e00\u4e2a\u6709\u6548\u7684\u65b9\u6cd5\u662f\u5728\u54cd\u5e94\u4e2d\u6dfb\u52a0\u5f15\u7528\uff0c\u8ba9\u7528\u6237\u770b\u5230\u4fe1\u606f\u6765\u6e90\u3002\u6211\u4eec\u53ef\u4ee5\u4f7f\u7528\u7a0d\u5fae\u4e0d\u540c\u7248\u672c\u7684RetrievalQA\u94fe\uff0c\u79f0\u4e3aRetrievalQAWithSourcesChain\uff0c\u6765\u5b9e\u73b0\u8fd9\u4e00\u70b9\u3002<\/p>\n<p>In[23]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">ini<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-ini code-block-extension-codeShowNum\" lang=\"ini\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">from langchain.chains import RetrievalQAWithSourcesChain<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"><span class=\"hljs-attr\">qa_with_sources<\/span> = RetrievalQAWithSourcesChain.from_chain_type(<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"4\">    <span class=\"hljs-attr\">llm<\/span>=llm,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"5\">    <span class=\"hljs-attr\">chain_type<\/span>=<span class=\"hljs-string\">\"stuff\"<\/span>,<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"6\">    <span class=\"hljs-attr\">retriever<\/span>=vectorstore.as_retriever()<\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"7\">)<\/span>\n<\/code><\/pre>\n<p>In[24]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">scss<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-scss code-block-extension-codeShowNum\" lang=\"scss\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\"><span class=\"hljs-built_in\">qa_with_sources<\/span>(query)<\/span>\n<\/code><\/pre>\n<p>Out[24]:<\/p>\n<pre><\/div><div class=\"code-block-extension-headerRight\"><span class=\"code-block-extension-lang\">vbnet<\/span><div class=\"code-block-extension-copyCodeBtn\">\u590d\u5236\u4ee3\u7801<\/div><\/div><\/div><code class=\"hljs language-vbnet code-block-extension-codeShowNum\" lang=\"vbnet\"><span class=\"code-block-extension-codeLine\" data-line-num=\"1\">{<span class=\"hljs-comment\">'question': 'who was Benito Mussolini?',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"2\"> <span class=\"hljs-comment\">'answer': 'Benito Mussolini was an Italian politician and journalist who was the Prime Minister of Italy from 1922 until 1943. He was the leader of the National Fascist Party and invented the ideology of Fascism. He became dictator of Italy by the end of 1927 and was friends with German dictator Adolf Hitler. Mussolini attacked Greece and failed to conquer it. He was removed by the Great Council of Fascism in 1943 and was executed by a partisan on April 28, 1945. After the war, several Neo-Fascist movements have had success in Italy, the most important being the Movimento Sociale Italiano. His granddaughter Alessandra Mussolini has outspoken views similar to Fascism. n',<\/span><\/span>\n<span class=\"code-block-extension-codeLine\" data-line-num=\"3\"> <span class=\"hljs-comment\">'sources': 'https:\/\/simple.wikipedia.org\/wiki\/Benito%20Mussolini, https:\/\/simple.wikipedia.org\/wiki\/Fascism'}<\/span><\/span>\n<\/code><\/pre>\n<p>\u73b0\u5728\uff0c\u6211\u4eec\u4e0d\u4ec5\u56de\u7b54\u4e86\u63d0\u51fa\u7684\u95ee\u9898\uff0c\u8fd8\u5305\u62ec\u4e86LLM\u4f7f\u7528\u7684\u8fd9\u4e9b\u4fe1\u606f\u7684\u6765\u6e90\u3002<\/p>\n<p>\u6211\u4eec\u5df2\u7ecf\u5b66\u4f1a\u4e86\u901a\u8fc7\u4f7f\u7528\u77e2\u91cf\u6570\u636e\u5e93\u4f5c\u4e3a\u6211\u4eec\u7684\u77e5\u8bc6\u5e93\u6765\u4e3a\u5927\u578b\u8bed\u8a00\u6a21\u578b\u63d0\u4f9b\u6e90\u77e5\u8bc6\u7684\u57fa\u7840\u3002\u5229\u7528\u8fd9\u4e00\u70b9\uff0c\u6211\u4eec\u53ef\u4ee5\u9f13\u52b1LLM\u7684\u54cd\u5e94\u51c6\u786e\u6027\uff0c\u4fdd\u6301\u6e90\u77e5\u8bc6\u7684\u66f4\u65b0\uff0c\u5e76\u901a\u8fc7\u5728\u6bcf\u4e2a\u7b54\u6848\u4e2d\u63d0\u4f9b\u5f15\u7528\u6765\u63d0\u9ad8\u5bf9\u6211\u4eec\u7cfb\u7edf\u7684\u4fe1\u4efb\u3002<\/p>\n<p>\u6211\u4eec\u5df2\u7ecf\u770b\u5230LLM\u548c\u77e5\u8bc6\u5e93\u5728\u5927\u578b\u4ea7\u54c1\u4e2d\u914d\u5bf9\u4f7f\u7528\uff0c\u4f8b\u5982\u5fc5\u5e94\u7684AI\u641c\u7d22\u3001Google Bard\u548cChatGPT\u63d2\u4ef6\u3002\u6beb\u65e0\u7591\u95ee\uff0cLLM\u7684\u672a\u6765\u4e0e\u9ad8\u6027\u80fd\u3001\u53ef\u6269\u5c55\u548c\u53ef\u9760\u7684\u77e5\u8bc6\u5e93\u7d27\u5bc6\u76f8\u8fde\u3002<\/p>","protected":false},"excerpt":{"rendered":"<p>\u5927\u578b\u8bed\u8a00\u6a21\u578b\uff08LLMs\uff09\u5b58\u5728\u6570\u636e\u65b0\u9c9c\u5ea6\u95ee\u9898\u3002\u5373\u4f7f\u662f\u50cfGPT-4\u8fd9\u6837\u6700\u5f3a\u5927\u7684\u6a21\u578b\uff0c\u4e5f\u4e0d\u4e86\u89e3\u6700\u8fd1\u7684\u4e8b\u4ef6\u3002 \u6839\u636eLLMs\u7684\u89c6\u89d2\uff0c\u4e16\u754c\u4eff\u4f5b\u505c\u6ede\u5728\u67d0\u4e2a\u65f6\u95f4\u70b9\u3002\u5b83\u4eec\u53ea\u77e5\u9053\u4e16\u754c\u662f\u5982\u4f55\u5728\u5b83\u4eec\u7684\u8bad\u7ec3\u6570\u636e\u4e2d\u5448\u73b0\u7684\u3002 \u8fd9\u5bf9<\/p>\n","protected":false},"author":1,"featured_media":7820,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"rank_math_title":"","rank_math_description":"","rank_math_focus_keyword":"","views":"5","footnotes":""},"categories":[3],"tags":[126,127,128,129,136],"collection":[],"class_list":["post-1359","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-fenlei2","tag-gpt","tag-ai","tag-128","tag-129","tag-136"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts\/1359","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/comments?post=1359"}],"version-history":[{"count":0,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/posts\/1359\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/media\/7820"}],"wp:attachment":[{"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/media?parent=1359"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/categories?post=1359"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/tags?post=1359"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/www.nicekj.com\/nicekj2024\/wp\/v2\/collection?post=1359"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}