Top suggestions for Use Cases Large Language Models Inference by Structured Sparsity |
+by+skipping+several+layers+of+computations+through+elaborate+heuristics+or+additional+predictors.+However%252C+in+the+decoding+process+of+existing+approaches%252C+different+samples+are+assigned+different+computational+budgets%252C+which+cannot+guarantee+a+stable+and+precise+acceleration+effect.+Furthermore%252C+existing+approaches+generally+skip+multiple+contiguous+layers+at+the+bottom+or+top+of+the+layers%252C+leading+to+a+drastic+change+in+the+model%27s+layer-wise+representations%252C+and+thus+a+consequent+performance+degeneration.+Therefore%252C+we+propose+a+Unified+Layer+Skipping+strategy%252C+which+selects+the+number+of+layers+to+skip+computation+based+solely+on+the+target+speedup+ratio%252C+and+then+skips+the+corresponding+number+of+intermediate+layer+computations+in+a+balanced+manner.+Since+the+Unified+Layer+Skipping+strategy+is+independent+of+input+samples%252C+it+naturally+supports+popular+acceleration+techniques+such+as+batch+decoding+and+KV+caching%252C+thus+demonstrating+more+practicality+for+real-world+applications.+Experimental+results+on+two+common+tasks%252C+i.e.%252C+machine+translation+and+text+summarization%252C+indicate+that+given+a+target+speedup+ratio%252C+the+Unified+Layer+Skipping+strategy+significantly+enhances+both+the+inference+performance+and+the+actual+model+throughput+over+existing+dynamic+approaches.&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.42c9dcc9f3cc8cc7434732ddde42dc5b%3frik%3dBwa1H5g5U3Kv5Q%26pid%3dImgRaw%26r%3d0&exph=630&expw=1200&q=Use+Cases+Large+Language+Models+Inference+by+Structured+Sparsity&FORM=IRPRST&ck=CE87E8DA14E42872303E42899A54C320&selectedIndex=9&itb=0)
- 850×1100
Some results have been hidden because they may be inaccessible to you.Show inaccessible results
Top suggestions for Use Cases Large Language Models Inference by Structured Sparsity |
Feedback
